[2025-05-31 02:58:27.541060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.548829] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:27.555962] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.566221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.570542] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81060][TimerWK0_ServerTracerTim][T0][YB42AC1087A6-0006366547BA5925-0-0] [lt=37] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:27.578203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.589054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.590342] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A62-0-0] [lt=28] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:27.590381] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591452] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591476] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591563] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591594] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.591900] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:27.600950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.611228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.622344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.632927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.637294] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:27.637401] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3555, avg_group_log_size=355) [2025-05-31 02:58:27.637426] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3981) [2025-05-31 02:58:27.637441] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:27.639605] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4556, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:27.639632] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3555) [2025-05-31 02:58:27.639772] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=537, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:27.640022] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:27.643022] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [8165][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F77-0-0] [lt=129] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=2760, avg_log_gen_to_submit_time=2856, avg_log_submit_to_flush_time=5872, avg_log_submit_to_slide_time=7186) [2025-05-31 02:58:27.643387] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3555, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:27.643436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.654035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.668385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.679166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.689830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.703740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.706603] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:27.706642] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-05-31 02:58:27.706653] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:27.706670] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:27.706685] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525355888636, flush_io_finished_ret_:0, flush_io_finished_round_:19875, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:27.714498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.714812] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:27.718438] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:27.718557] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, norm_iops:2, bw:40525, [delay/us]:prepare:11, schedule:93, submit:659, rt:2891, total:3655) [2025-05-31 02:58:27.718580] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:27.718693] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=96] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40523, [delay/us]: prepare:11, schedule:93, submit:659, device:2891, total:3655) [2025-05-31 02:58:27.718707] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:27.718733] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4052, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:405, total:421) [2025-05-31 02:58:27.718758] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:27.718820] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4052, [delay/us]: prepare:0, schedule:0, submit:600000000, device:405, total:421) [2025-05-31 02:58:27.718841] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:27.718911] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:27.718920] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:27.718929] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:27.718938] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:27.718947] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=186kB/s, net_out=192kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:27.718991] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399403][2nd=2348]}) [2025-05-31 02:58:27.725279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.725417] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4464-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:27.735691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.746368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.749858] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:27.756967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.764542] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83086][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA53-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:27.770140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.780537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.791818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.803059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=144][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.813254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.817306] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678307817301, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006321, add_bytes: 0 [2025-05-31 02:58:27.823714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.830044] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678307329190], gts:1748678307365261000, latest_srr:[mts=1748678307830017]}) [2025-05-31 02:58:27.834051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.840950] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=28][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09bd4473e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, prev=this=0x2b09bd421a48 trans_version={val:1748677707647533000, v:0} scn={val:1748677707647533000, v:0} tx_id={txid:10037785} prev=0x2b09a8011858 next=(nil) modify_count=9 acc_checksum=1798317612 version=1748674829456371 type=0 flag=4 snapshot_barrier=1748678305616133000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:18587} write_epoch=3, snapshot_version={val:1748678307834249000, v:0}, *this={this=0x2b09bd408ac8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b09bd421a48 latest_compact_node=0x2b097c77da18 max_trans_version={val:1748677707647533000, v:0} max_trans_id=10037785 max_elr_trans_version={val:1748677707647533000, v:0} max_elr_trans_id=10037785 latest_compact_ts=1748676507167583 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677707647533000, v:0} min_modify_scn={val:1748675247608429000, v:0}}) [2025-05-31 02:58:27.841017] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=65][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09bd4473e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678337735316, tablet_id:{id:444}, table_iter:0x2b091bf4f6d8, table_version:1748168834207416, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678337735316, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678307834249000, v:0}, tx_id:{txid:10038481}, scn:{branch:0, seq:6165}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038481}, tx_desc:{this:0x2b094e865ba0, tx_id:{txid:10038481}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:3222526617, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678307839505, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678307864091, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:6168}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:27.841112] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038481}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674829456371, seq_no:{branch:0, seq:6168}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-05-31 02:58:27.841131] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"BIGINT UNSIGNED":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168834207416 max_table_version=1748168834207416 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038481} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:27.841161] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:27.841175] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2b09c620c780, local_datums:0x2b09c620c780}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c1f4, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c1fc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=2:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c204, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c211, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c219, hex: 0FE05DE569360600, int: 1748678307864591}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c221, hex: 787FEBF3A78D4418, int: 1748678307817947000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c229, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:27.841252] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:45) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=51] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=60620, insert_dup_release_count=60620, col_map_alloc_count=0, col_map_release_count=0) [2025-05-31 02:58:27.841279] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=16][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:27.841315] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=8][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09bd447578 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, prev=this=0x2b09bd421598 trans_version={val:1748677707647533000, v:0} scn={val:1748677707647533000, v:0} tx_id={txid:10037785} prev=0x2b09a8011678 next=(nil) modify_count=9 acc_checksum=119739084 version=1748674829456371 type=0 flag=4 snapshot_barrier=1748678305616133000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:18587} write_epoch=3, snapshot_version={val:1748678307834249000, v:0}, *this={this=0x2b09bd408c78 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b09bd421598 latest_compact_node=0x2b097c77d808 max_trans_version={val:1748677707647533000, v:0} max_trans_id=10037785 max_elr_trans_version={val:1748677707647533000, v:0} max_elr_trans_id=10037785 latest_compact_ts=1748676507167569 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677707647533000, v:0} min_modify_scn={val:1748675247608429000, v:0}}) [2025-05-31 02:58:27.841347] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=31][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09bd447578 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678337735316, tablet_id:{id:444}, table_iter:0x2b091bf4f6d8, table_version:1748168834207416, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678337735316, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678307834249000, v:0}, tx_id:{txid:10038481}, scn:{branch:0, seq:6165}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038481}, tx_desc:{this:0x2b094e865ba0, tx_id:{txid:10038481}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:3222526617, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678307839505, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678307864091, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:6168}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:27.841415] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=66][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038481}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674829456371, seq_no:{branch:0, seq:6168}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-05-31 02:58:27.841428] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"BIGINT UNSIGNED":2},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168834207416 max_table_version=1748168834207416 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038481} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:27.841449] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:27.841458] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2b09c620c780, local_datums:0x2b09c620c780}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2b4, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2bc, hex: 0200000000000000, int: 2}, col_id=2:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c2c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2d1, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2d9, hex: 0FE05DE569360600, int: 1748678307864591}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2e1, hex: 200711D1A78D4418, int: 1748678307233204000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c2e9, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:27.841595] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=68][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:27.841657] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09bd447708 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, prev=this=0x2b09bd421728 trans_version={val:1748677707647533000, v:0} scn={val:1748677707647533000, v:0} tx_id={txid:10037785} prev=0x2b09a8011718 next=(nil) modify_count=9 acc_checksum=705577150 version=1748674829456371 type=0 flag=4 snapshot_barrier=1748678305616133000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:18587} write_epoch=3, snapshot_version={val:1748678307834249000, v:0}, *this={this=0x2b09bd408e28 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b09bd421728 latest_compact_node=0x2b097c77d8b8 max_trans_version={val:1748677707647533000, v:0} max_trans_id=10037785 max_elr_trans_version={val:1748677707647533000, v:0} max_elr_trans_id=10037785 latest_compact_ts=1748676507167574 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677707647533000, v:0} min_modify_scn={val:1748675247608429000, v:0}}) [2025-05-31 02:58:27.841731] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=73][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09bd447708 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678337735316, tablet_id:{id:444}, table_iter:0x2b091bf4f6d8, table_version:1748168834207416, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678337735316, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678307834249000, v:0}, tx_id:{txid:10038481}, scn:{branch:0, seq:6165}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038481}, tx_desc:{this:0x2b094e865ba0, tx_id:{txid:10038481}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:3222526617, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678307839505, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678307864091, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:6168}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:27.841801] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=69][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038481}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674829456371, seq_no:{branch:0, seq:6168}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-05-31 02:58:27.841814] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168834207416 max_table_version=1748168834207416 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038481} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:27.841896] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=80][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:27.841908] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2b09c620c780, local_datums:0x2b09c620c780}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c374, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c37c, hex: 0300000000000000, int: 3}, col_id=2:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c384, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c391, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c399, hex: 0FE05DE569360600, int: 1748678307864591}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c3a1, hex: 9011F7EDA78D4418, int: 1748678307718042000}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c3a9, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:27.841975] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=41][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:27.842012] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=10][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09bd447898 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, prev=this=0x2b09bd4218b8 trans_version={val:1748677707647533000, v:0} scn={val:1748677707647533000, v:0} tx_id={txid:10037785} prev=0x2b09a80117b8 next=(nil) modify_count=9 acc_checksum=2041748456 version=1748674829456371 type=0 flag=4 snapshot_barrier=1748678305616133000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:18587} write_epoch=3, snapshot_version={val:1748678307834249000, v:0}, *this={this=0x2b09bd408fd8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b09bd4218b8 latest_compact_node=0x2b097c77d968 max_trans_version={val:1748677707647533000, v:0} max_trans_id=10037785 max_elr_trans_version={val:1748677707647533000, v:0} max_elr_trans_id=10037785 latest_compact_ts=1748676507167579 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677707647533000, v:0} min_modify_scn={val:1748675247608429000, v:0}}) [2025-05-31 02:58:27.842043] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=30][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09bd447898 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038481} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674829456371 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:64} seq_no={branch:0, seq:6168} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678337735316, tablet_id:{id:444}, table_iter:0x2b091bf4f6d8, table_version:1748168834207416, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678337735316, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678307834249000, v:0}, tx_id:{txid:10038481}, scn:{branch:0, seq:6165}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038481}, tx_desc:{this:0x2b094e865ba0, tx_id:{txid:10038481}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:3222526617, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678307839505, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678307864091, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:6168}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:27.842110] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=66][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038481}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674829456371, seq_no:{branch:0, seq:6168}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:7}) [2025-05-31 02:58:27.842123] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=12][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"BIGINT UNSIGNED":4},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168834207416 max_table_version=1748168834207416 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038481} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:27.842144] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=19][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:27.842152] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=444, col_descs=[cnt:7, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:16, datums:0x2b09c620c780, local_datums:0x2b09c620c780}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c434, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c43c, hex: 0400000000000000, int: 4}, col_id=2:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c444, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c451, hex: 420B000000000000, int: 2882}, col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c459, hex: 0FE05DE569360600, int: 1748678307864591}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c461, hex: FFFFFFFFFFFFFF3F, int: 4611686018427387903}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c469, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:27.842212] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=38][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:27.842349] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=9] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"172.16.135.30:45422"}) [2025-05-31 02:58:27.842973] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37E6-0-0] [lt=31] [APPLY STAT CB APPEND COST TIME](cur_stat_count=43, stat_interval=5000000, avg (count/cost)=166, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:27.843008] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37E6-0-0] [lt=35] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=43, stat_interval=5000000, avg (count/cost)=7040, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:27.843016] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37E6-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=43, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:27.843086] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37E6-0-0] [lt=69] [APPLY STAT CB EXECUTE TIME](cur_stat_count=43, stat_interval=5000000, avg (count/cost)=-1479, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:27.843094] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37E6-0-0] [lt=8] [APPLY STAT CB TOTAL TIME](cur_stat_count=43, stat_interval=5000000, avg (count/cost)=5727, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:27.845081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.846620] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 03:58:27.848103] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [12787][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:27.848399] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83100][T1_L0_G0][T1][YB42AC1087A7-0006366543FE05F2-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678307847377000, start_id=1748678307847377000, end_id=1748678307847377001) [2025-05-31 02:58:27.848495] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:58:27.849397] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83078][T1_L0_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=13] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:12855}, to_seq_no={branch:0, seq:6166}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168834207416 max_table_version=1748168834207416 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038481} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:27.849500] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83078][T1_L0_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=55] rpc statistics(total_trans_msg_count=17, total_batch_msg_count=0) [2025-05-31 02:58:27.849514] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:148) [83078][T1_L0_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=12] transaction rpc statistics(total_rt=1479, total_process=9, avg_rt=147) [2025-05-31 02:58:27.853081] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=16] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"172.16.135.30:45386"}) [2025-05-31 02:58:27.855834] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=81] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"172.16.135.30:45366"}) [2025-05-31 02:58:27.857325] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [83067][T1_L5_G0][T1][YB42AC10871E-000636654D484B6E-0-0] [lt=94] sql stat record not found(ret=-5138, key={sql_id:"7E1EE37EACA2C780A528E1CE2BBEA33", plan_hash:8894128478548303855, source_addr:"172.16.135.30:45398"}) [2025-05-31 02:58:27.858256] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC10871E-000636654D484B6F-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678307857269000, start_id=1748678307857269000, end_id=1748678307857269001) [2025-05-31 02:58:27.858504] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83059][T1_L0_G0][T1][YB42AC10871E-000636654D484B6F-0-0] [lt=30] transaction rpc statistics(total_rt=14155, total_process=47, avg_rt=294) [2025-05-31 02:58:27.858811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.860409] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:27.860437] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=6899, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:27.868520] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37EB-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525357050689, cache_obj->added_lc()=false, cache_obj->get_object_id()=53565, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:27.872388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.882607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.889515] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1748678307889510, bytes: 7125314936, bw: 0.220433 MB/s, add_ts: 1005206, add_bytes: 232344 [2025-05-31 02:58:27.890266] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:27.894030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.905719] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.916629] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:27.917015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.918465] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=107] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-05-31 02:58:27.920343] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:27.927461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.940310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.950740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.953525] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:27.964237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=191][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.974552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.984755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:27.989938] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205B-0-0] [lt=72] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:27.989981] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205B-0-0] [lt=34] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=95) [2025-05-31 02:58:27.998758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=144][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.007677] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:28.009263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.019527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.032037] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=18] gc stale ls task succ [2025-05-31 02:58:28.032835] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.036388] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:28.039308] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678308038095000, start_id=1748678308038095000, end_id=1748678308038095001) [2025-05-31 02:58:28.043773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=142][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.045549] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678307633403000, v:0}, new_max_version={val:1748678307633403000, v:0}, last_min_version={val:1748678307436273000, v:0}, last_max_version={val:1748678307436273000, v:0}, total_time_us=9001, delta_us=412143) [2025-05-31 02:58:28.045609] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678307633403000, v:0}, min_version={val:1748678307633403000, v:0}, max_version={val:1748678307633403000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:28.046980] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:28.054952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.065190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.075163] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:28.075580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.085925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.096175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.106597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.116958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.117004] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:28.127237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.137488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.149226] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [12787][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F7E-0-0] [lt=2101] init point thread id with(&point=0x55c0908cf080, idx_=499, point=[thread id=12787, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=12787) [2025-05-31 02:58:28.149540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.154228] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:28.160197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.170583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.185909] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AD6-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:28.186495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.198094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.208440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.218748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.229114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.241487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.242767] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:28.242797] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=29] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:28.242959] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1094-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:28.242989] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1094-0-0] [lt=26] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:28.243211] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F7F-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5907) [2025-05-31 02:58:28.253560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.267184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.277837] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.288357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=173][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.290623] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5926-0-0] [lt=28] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678303531438, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:28.290676] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5926-0-0] [lt=50] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678303534099, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:28.290701] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5926-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678303543449, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678303543456, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:28.298701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.309120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.319719] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:28.319801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.330074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.333193] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:28.340482] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.344508] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/11:60491 disk:52K/11:60491 t1m2:52K/11:60491 t1m2g0:52K/11:60491 |q0: 0 |q1: 0 [2025-05-31 02:58:28.351959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.356777] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:28.358683] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.358989] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.359024] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.360019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=93] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.363146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.373352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.383750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.394195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.404683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=174][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.416596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.419015] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:58:28.424548] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:28.427381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=163][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.437919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=238][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.441951] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8A6-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.724449205398559570e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.590689397536605611e+03) [2025-05-31 02:58:28.453259] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C0-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:28.455293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.466622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.474826] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A19-0-0] [lt=111][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525357656995, cache_obj->added_lc()=false, cache_obj->get_object_id()=53566, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:28.479001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.489214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.499714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.512333] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.522176] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:28.524596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.534945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.546452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.556704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.557799] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:28.572065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=181][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.583126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.590376] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.590438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.591432] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.591475] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=65] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.591530] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.591560] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.592103] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:28.592137] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:28.592171] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:28.592279] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:28.592308] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:28.595034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.603957] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=40705) [2025-05-31 02:58:28.605190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.615412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.626837] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=330][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.638163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=164][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.644662] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=107, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:28.644697] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5907) [2025-05-31 02:58:28.644706] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=5785, avg_group_log_size=525) [2025-05-31 02:58:28.644723] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:28.647032] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2322, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:28.647154] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=119] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5785) [2025-05-31 02:58:28.647185] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=234, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:28.647382] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=41, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:28.651511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.651968] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [6894][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F83-0-0] [lt=61] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=459, avg_log_gen_to_submit_time=590, avg_log_submit_to_flush_time=2759, avg_log_submit_to_slide_time=4389) [2025-05-31 02:58:28.652388] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5785, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 03:58:28.653007] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [13069][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:28.655080] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:28.656515] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD34-0-0] [lt=25] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678308653970, keepalive_ts:-1}) [2025-05-31 02:58:28.656549] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD34-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678308653970, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:28.665343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.666349] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=26] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:28.666408] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=40] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=963) [2025-05-31 02:58:28.676127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.686320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.696919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.707457] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:28.707511] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-05-31 02:58:28.707523] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:28.707538] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:28.707554] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525356888901, flush_io_finished_ret_:0, flush_io_finished_round_:19876, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:28.707616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.718061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.722308] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:28.728946] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4465-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:28.729092] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:28.729149] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, norm_iops:3, bw:52687, [delay/us]:prepare:12, schedule:53, submit:194, rt:1962, total:2222) [2025-05-31 02:58:28.729167] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:28.729282] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=99] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:52690, [delay/us]: prepare:12, schedule:53, submit:194, device:1962, total:2222) [2025-05-31 02:58:28.729298] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:28.729325] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4053, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:336, total:351) [2025-05-31 02:58:28.729351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:28.729382] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4053, [delay/us]: prepare:0, schedule:0, submit:600000000, device:336, total:351) [2025-05-31 02:58:28.729400] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:28.729414] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:28.729423] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:28.729432] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:28.729440] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:28.729450] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=230kB/s, net_out=164kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:28.729469] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399404][2nd=2925]}) [2025-05-31 02:58:28.729531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.739760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.750047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.759101] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:28.760556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.771069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=150][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.781825] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA54-0-0] [lt=100][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:28.784436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.794653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.802037] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:58:28.802111] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=61] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:58:28.802140] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=18] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:28.804960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.810001] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:28.810186] WDIAG read_multi_row (ob_table_access_helper.h:238) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:28.812443] INFO delete_row (ob_table_access_helper.h:339) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678308810206) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678308810206) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:58:28.815322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.817521] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678308817515, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000214, add_bytes: 0 [2025-05-31 02:58:28.826003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.834189] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678308833514000, start_id=1748678308833514000, end_id=1748678308833514001) [2025-05-31 02:58:28.836330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.846709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.857209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=327][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.867459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.877978] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81743][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=31] clear dtl interm result cost(us)(clear_cost=3576, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:58:28.878305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.885632] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37ED-0-0] [lt=83][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525358067799, cache_obj->added_lc()=false, cache_obj->get_object_id()=53567, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:28.888587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.890654] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=29] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:28.894898] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1748678308894893, bytes: 7125515013, bw: 0.189787 MB/s, add_ts: 1005383, add_bytes: 200077 [2025-05-31 02:58:28.899975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.910350] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.922315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.922357] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:58:28.922688] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:58:28.922957] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:28.927080] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:28.932582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.943973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.954680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.959585] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:28.961124] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=377484, clean_num=31457) [2025-05-31 02:58:28.964960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.975231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.985930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:28.999980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.011540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.013213] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=41] ====== check clog disk timer task ====== [2025-05-31 02:58:29.013246] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:29.022133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.032455] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=42] current monitor number(seq_=-1) [2025-05-31 02:58:29.032576] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=31] gc stale ls task succ [2025-05-31 02:58:29.032759] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.036969] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:29.043061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.048953] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:29.049083] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:196) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] skip repeat study(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678249999412, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677707084975001, v:0}, is_inited:true}, current_timestamp=1748678309049079, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-05-31 02:58:29.051145] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=165] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678152214536000, snapshot_type=1, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:58:29.051171] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009655406000, snapshot_type=1, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:58:29.051181] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678190211897000, snapshot_type=1, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:58:29.051191] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677990551999001, snapshot_type=1, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:58:29.051199] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185755930000, snapshot_type=1, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:58:29.051208] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307817947000, snapshot_type=1, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:58:29.051216] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678151786499001, snapshot_type=2, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:58:29.051224] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009054758001, snapshot_type=2, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:58:29.051233] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678189609335001, snapshot_type=2, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:58:29.051241] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677989766058002, snapshot_type=2, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:58:29.051249] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185265507000, snapshot_type=2, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:58:29.051257] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307233204000, snapshot_type=2, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:58:29.051265] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678151908197000, snapshot_type=3, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:58:29.051273] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009457625000, snapshot_type=3, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:58:29.051364] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678189822651000, snapshot_type=3, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:58:29.051380] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677990139537000, snapshot_type=3, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:58:29.051388] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185611745000, snapshot_type=3, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:58:29.051397] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307718042000, snapshot_type=3, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:58:29.051405] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:58:29.051466] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=56] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:58:29.051477] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:58:29.051485] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:58:29.051493] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:58:29.051501] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:58:29.051779] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:625) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update global reserved snapshot(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678249999412, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677707084975001, v:0}, is_inited:true}, global_reserved_snapshot_={val:1748677707084975001, v:0}, reserved_snapshot={val:1748677989766058002, v:0}) [2025-05-31 02:58:29.051803] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:559) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] multi-version garbage collector refresh successfully(this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678249999412, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1748677989766058002, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1748677990562207, reserved_addr:"172.16.135.168:2882", is_this_server_disabled:false, status:0}) [2025-05-31 02:58:29.051821] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:231) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678309051820, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, current_timestamp=1748678309049248, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-05-31 02:58:29.051832] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:266) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] skip repeat reclaim(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678309051820, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, current_timestamp=1748678309051832, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-05-31 02:58:29.053224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.054245] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:29.054275] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6574, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:29.058579] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678308833514000, v:0}, new_max_version={val:1748678308833514000, v:0}, last_min_version={val:1748678308644190001, v:0}, last_max_version={val:1748678308644190001, v:0}, total_time_us=13138, delta_us=225062) [2025-05-31 02:58:29.058621] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678308833514000, v:0}, min_version={val:1748678308833514000, v:0}, max_version={val:1748678308833514000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:29.058647] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:29.063438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.073644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.084203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.096265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.106519] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.116755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.124824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:29.127956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.138258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.145619] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678309046930000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678309145563}, version_delta=98686) [2025-05-31 02:58:29.149255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.159957] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.162156] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=74] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:29.173500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.184066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.194375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.205247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.215538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.222926] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0CDA-0-0] [lt=36] handle gts request statistics(total_rt=1220, total_cnt=182, avg_rt=6.666666666666666963e+00, avg_cnt=1.819999999999999929e+01) [2025-05-31 02:58:29.226040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.236921] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.247199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.248999] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=33] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:36K/9:90261 disk:36K/9:90261 t1m2:36K/9:90261 t1m2g0:36K/9:90261 |q0: 0 |q1: 0 [2025-05-31 02:58:29.250602] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3433) [2025-05-31 02:58:29.259058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.261681] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=117] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:29.269362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.279663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.290246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=363][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.300647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.313847] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.324320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=204][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.325103] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=96] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:29.334631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.336089] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=61] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678309336047], gts:1748678309335373000, latest_srr:[mts=1748678309336047]}) [2025-05-31 02:58:29.336129] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:29.336187] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:29.345067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.355400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.359587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.359699] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.359735] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.359762] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.360360] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=98] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:29.362790] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:29.364040] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:455) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=40] skip defragment task(ret=0, need_defragment=false) [2025-05-31 02:58:29.364074] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:497) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=31] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=11) [2025-05-31 02:58:29.364086] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:515) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=13] successfully defragment data blocks(rewrite_cnt=0) [2025-05-31 02:58:29.364097] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=349942, total_skip_cnt=6068, not_in_mem_tablet_cnt=516, invalid_attr_tablet_cnt=0) [2025-05-31 02:58:29.365684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.370783] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=31] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-05-31 02:58:29.375959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.388234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.398525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.408823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.421936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=169][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.423243] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-05-31 02:58:29.427295] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=51] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:29.432232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.442640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.446120] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] no need to update min/max version(new_version={val:1748678309046930000, v:0}, min_version={val:1748678309046930000, v:0}, max_version={val:1748678309046930000, v:0}) [2025-05-31 02:58:29.455745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.458308] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83075][T1_L0_G0][T1][YB42AC1087A8-000636672F6E3FFA-0-0] [lt=22] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=283, Holding Memory(MB)=134, Tenant Remain Memory(MB)=249, Usable Remain Memory(MB)=149, Last Update Limit Timestamp=1748678309457083, Is Updated=true) [2025-05-31 02:58:29.458369] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83075][T1_L0_G0][T1][YB42AC1087A8-000636672F6E3FFA-0-0] [lt=57] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.676449251174926758e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.678656597848224465e+03) [2025-05-31 02:58:29.458716] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83075][T1_L0_G0][T1][YB42AC1087A8-000636672F6E3FFA-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1748678309457083000, start_id=1748678309457083000, end_id=1748678309457083001) [2025-05-31 02:58:29.461227] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [13069][T1_L0_G1][T1][YB42AC1087A7-00063665517F53C3-0-0] [lt=27] init point thread id with(&point=0x55c0908d3700, idx_=781, point=[thread id=13069, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=13069) [2025-05-31 02:58:29.466394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.476666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.488184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.499240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.509836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.528816] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.529043] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:29.540294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.546257] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678309247461000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=298785, in_cluster_service=true, cluster_version={val:1748678309046930000, v:0}, min_cluster_version={val:1748678309046930000, v:0}, max_cluster_version={val:1748678309046930000, v:0}, get_cluster_version_err=0, cluster_version_delta=499316, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=194943, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=300491488, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:29.546369] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95] no need to update min/max version(new_version={val:1748678309046930000, v:0}, min_version={val:1748678309046930000, v:0}, max_version={val:1748678309046930000, v:0}) [2025-05-31 02:58:29.552195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.562443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.566531] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:29.572724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.585335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.590485] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.590597] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.593747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.593791] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.593818] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.593844] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.594166] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:29.596095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.601018] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=31] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:29.608372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.618627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.624588] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83091][T1_L0_G0][T1][YB42AC108714-000636671E7D5D4E-0-0] [lt=132] get number(ret=0, service_type_=0, range=1, base_id=1748678309621662000, start_id=1748678309621662000, end_id=1748678309621662001) [2025-05-31 02:58:29.631149] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.644237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.646819] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83076][T1_L0_G0][T1][YB42AC10871E-0006366542D8A6FF-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1748678309640604000, start_id=1748678309640604000, end_id=1748678309640604001) [2025-05-31 02:58:29.647138] INFO [LIB] stat (utility.h:1211) [83076][T1_L0_G0][T1][YB42AC10871E-0006366542D8A6FF-0-0] [lt=120] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=102, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:29.647188] INFO [LIB] stat (utility.h:1211) [83076][T1_L0_G0][T1][YB42AC10871E-0006366542D8A6FF-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:29.649180] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4366) [2025-05-31 02:58:29.653859] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2730, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:29.653904] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4366) [2025-05-31 02:58:29.653931] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1087, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:29.654118] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4955, avg_group_log_size=450) [2025-05-31 02:58:29.654168] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=62, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:29.655511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.655553] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:29.658234] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [9040][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=91] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:29.658628] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665517F53C7-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=1722, avg_log_gen_to_submit_time=1836, avg_log_submit_to_flush_time=3878, avg_log_submit_to_slide_time=5209) [2025-05-31 02:58:29.658761] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4955, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:29.661634] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:29.665732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.676036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.687094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.697749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.698387] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678299693288, end_time=1748678309698383, dag_status={}) [2025-05-31 02:58:29.698426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=35] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698458] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698473] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698481] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698495] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:29.698512] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:58:29.698520] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:58:29.708747] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:29.708835] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=80] temporary file module use no blocks [2025-05-31 02:58:29.708904] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:29.708923] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:29.708937] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525357889769, flush_io_finished_ret_:0, flush_io_finished_round_:19877, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:29.709069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.719308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.732696] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4466-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:29.733057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.733925] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=94368, replace_num=15728) [2025-05-31 02:58:29.733946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:29.740443] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:29.740504] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44550, [delay/us]:prepare:12, schedule:229, submit:87, rt:2404, total:2733) [2025-05-31 02:58:29.740522] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:29.740608] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44551, [delay/us]: prepare:12, schedule:229, submit:87, device:2404, total:2733) [2025-05-31 02:58:29.740625] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:29.740652] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4050, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:360, total:375) [2025-05-31 02:58:29.740678] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:29.740709] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4050, [delay/us]: prepare:0, schedule:0, submit:600000000, device:360, total:375) [2025-05-31 02:58:29.740727] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:29.740741] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:29.740750] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:29.740758] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:29.740766] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:29.740775] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=135kB/s, net_out=89kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:29.740796] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399405][2nd=2502]}) [2025-05-31 02:58:29.743258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.753475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.762088] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=28] ObArchiveSender is running(thread_index=0) [2025-05-31 02:58:29.763701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.767334] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:29.773991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.784165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.791017] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] BlockGCTimerTask success(ret=0, cost_time_us=21, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:58:29.794735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.799369] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA55-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:29.804976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.815167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.824914] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=487] PNIO [ratelimit] time: 1748678309824908, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007393, add_bytes: 0 [2025-05-31 02:58:29.825576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.836001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.849320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.859650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.865797] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=28] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19104587776, fast_delete_log_mode=false) [2025-05-31 02:58:29.869908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.881065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.891267] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:75, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183464487878, recv_hp_rpc_cnt:74786, recv_np_rpc_cnt:876054, recv_lp_rpc_cnt:0, recv_mysql_cnt:46890, recv_task_cnt:2468, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1229274 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9860, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:543024, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:7, nesting_worker_cnt:0, token_change:1748678309655548}{group_id:10, queue_size:0, recv_req_cnt:103733, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18320, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678293724148}{group_id:2, queue_size:0, recv_req_cnt:150627, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:100888, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:107699, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:271927, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=693 pcode=0x717:cnt=483 pcode=0x1502:cnt=228 pcode=0x710:cnt=161 pcode=0x523:cnt=99, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:58:29.892459] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1189] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:83850290, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:58:29.893262] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=801] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84356321, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:58:29.894086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=812][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.896434] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37EF-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525359078603, cache_obj->added_lc()=false, cache_obj->get_object_id()=53568, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:29.896956] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1748678309896952, bytes: 7125658607, bw: 0.136661 MB/s, add_ts: 1002059, add_bytes: 143594 [2025-05-31 02:58:29.897038] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:29.906451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.912617] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81426][pnio1][T0][YB42AC10870A-000636671AB945D3-0-0] [lt=24][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:1445140567, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678309934709, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1377656, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678309919604, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678309912612, enqueue_timestamp_:0, request_arrival_time_:1748678309912612, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:29.912724] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81426][pnio1][T0][YB42AC10870A-000636671AB945D3-0-0] [lt=86][errcode=-5150] deliver rpc request fail(&req=0x2b0959604098, ret=-5150) [2025-05-31 02:58:29.919137] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC10870A-000636671638EF08-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678309919050000, start_id=1748678309919050000, end_id=1748678309919050001) [2025-05-31 02:58:29.919425] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.926404] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:29.927674] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:29.930960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.934213] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:29.941604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.951897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.962103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.971258] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:29.972637] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.985354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:29.996137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.000111] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205C-0-0] [lt=352] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:30.000167] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205C-0-0] [lt=44] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=57) [2025-05-31 02:58:30.009488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.022422] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.033635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.033934] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=19] gc stale ls task succ [2025-05-31 02:58:30.039034] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:30.046725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.049168] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=49] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:30.056976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.058707] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:30.058737] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6185, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:30.062366] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678309648611001, v:0}, new_max_version={val:1748678309648611001, v:0}, last_min_version={val:1748678309648611000, v:0}, last_max_version={val:1748678309648611000, v:0}, total_time_us=11544, delta_us=413751) [2025-05-31 02:58:30.062465] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=94] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678309648611001, v:0}, min_version={val:1748678309648611001, v:0}, max_version={val:1748678309648611001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:30.067366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=186][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.077405] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81744][T1_TimerWK3_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=22] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:30.078167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.090647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.096769] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:30.101387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.110406] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83074][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA5F-0-0] [lt=61][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525359292574, cache_obj->added_lc()=false, cache_obj->get_object_id()=53569, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:30.111625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.122316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.132721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.134628] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=86] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:30.143421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.154254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.167080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.171909] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:30.177952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.189757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.200460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.213085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.224522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.234740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.245098] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=42] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:30.246363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.246590] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=34] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:30.246689] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=21] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678302084519000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:30.246727] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=36] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678302084519000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678302084519000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:58:30.247993] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=51] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:30.248087] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=91] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:30.248379] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1095-0-0] [lt=69] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:30.248406] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1095-0-0] [lt=24] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:30.255352] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4913) [2025-05-31 02:58:30.256796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.257638] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 03:58:30.259583] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [13661][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:30.259919] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:30.261953] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4131-0-0] [lt=82] get number(ret=0, service_type_=0, range=1, base_id=1748678310261435000, start_id=1748678310261435000, end_id=1748678310261435001) [2025-05-31 02:58:30.272458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.285295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.293062] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81060][TimerWK0_AutoLSLocRpc][T0][YB42AC1087A6-0006366547BA592A-0-0] [lt=45] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:58:30.295674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.303173] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F1-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525359485343, cache_obj->added_lc()=false, cache_obj->get_object_id()=53570, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:30.304921] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F3-0-0] [lt=35][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:30.305743] WDIAG get_refactored (ob_hashtable.h:1120) [83092][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F5-0-0] [lt=42][errcode=-4201] hashtable not init [2025-05-31 02:58:30.306163] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83106][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1748678310305234000, start_id=1748678310305234000, end_id=1748678310305234001) [2025-05-31 02:58:30.306675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.306935] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=39] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/11:167823 disk:52K/11:167823 t1m2:52K/11:167823 t1m2g0:52K/11:167823 |q0: 0 |q1: 0 [2025-05-31 02:58:30.321479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.327119] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [81744][T1_TimerWK3_CompactionRefre][T1][YB42AC1087A6-00063665431A60B0-0-0] [lt=39] InfoPoolResizeTask(cost_ts=8) [2025-05-31 02:58:30.332066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.334949] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:30.339959] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ts response task statistics(alloc_count=485594, free_count=242796) [2025-05-31 02:58:30.340012] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=754, get_gts_with_stc_cnt=33, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=33, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:30.340031] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:30.340098] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678309836963], gts:1748678309869157000, latest_srr:[mts=1748678310340045]}) [2025-05-31 02:58:30.342936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=307][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.353994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.358594] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.358638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.359248] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.359290] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.366754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.376782] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D5117-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525359558951, cache_obj->added_lc()=false, cache_obj->get_object_id()=53571, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:30.377082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.377422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:30.385764] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=1991, gc_interval=10000000) [2025-05-31 02:58:30.385824] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=34] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:58:30.385948] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=108] gc_check_member_list_ cost time(ret=0, time_us=149) [2025-05-31 02:58:30.385970] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=16] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:58:30.385985] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:58:30.386030] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=40] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:58:30.386042] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=13) [2025-05-31 02:58:30.386050] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-05-31 02:58:30.386058] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:58:30.386138] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:58:30.387318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.397517] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.407653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.417872] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.426587] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:30.428210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.430626] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:30.438490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.449356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.462263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.465367] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C1-0-0] [lt=24] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:30.470317] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=164637, log_service_ts=1748678310305234, min_tx_service_ts=4611686018427387, timestamp={val:1748678310305234000, v:0}) [2025-05-31 02:58:30.470420] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=93] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=64, Other_Error_Cnt=0, Submit_Succ_Cnt=37, last_scn="{val:1748678305430518000, v:0}", last_lsn={lsn:2137079158}, last_gts={val:1748678310455421000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:58:30.475090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.485498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.495735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.506047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.514518] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=35] ====== checkpoint timer task ====== [2025-05-31 02:58:30.515314] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=29] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:30.515363] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=48] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:30.515783] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678320364653000, latest_log_ts:{val:1748678303714352001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:30.515940] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=151] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678320364653000, latest_log_ts:{val:1748678303714352001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:30.516009] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:30.516025] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678303714352001, v:0}) [2025-05-31 02:58:30.516033] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:30.516041] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:30.516056] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:30.516077] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:30.516090] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:30.516111] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:30.516134] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:30.516144] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:30.516560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.527240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.532293] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=86] new mysql sessid created(conn.sessid_=3221735158, support_ssl=false) [2025-05-31 02:58:30.532407] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221735158, sess.client_addr_="172.16.135.166:43862") [2025-05-31 02:58:30.532428] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x2b08ff896450, session_id:3221735158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:141}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:30.532816] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81570][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:58:30.532838] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [81570][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=22] mysql login queue(mysql_queue->queue_.size()=1) [2025-05-31 02:58:30.533046] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE4-0-0] [lt=59] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221735158, conn.client_addr_port_=43862, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221735158, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221735158, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-000636654349FFE4-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82987, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418557937805912, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:245, fixup_index:-1, fixup_ash_buffer:{this:0x2b09651f7c58, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:58:30.535187] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:30.535302] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE4-0-0] [lt=161][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:58:30.535478] INFO [SERVER] process (obmp_connect.cpp:504) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE4-0-0] [lt=47] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221735158, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:58:30.536240] INFO [SERVER] process (obmp_init_db.cpp:44) [83080][T1_L0_G0][T1][YB42AC1087A6-00063665502A1388-0-0] [lt=41] init db(db_name=oceanbase) [2025-05-31 02:58:30.537963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.548173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.552122] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221735156, support_ssl=false) [2025-05-31 02:58:30.552185] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221735156, sess.client_addr_="172.16.135.166:43866") [2025-05-31 02:58:30.552203] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x2b08ff8971d0, session_id:3221735156, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:30.552584] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81570][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:58:30.552661] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDE-0-0] [lt=26] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221735156, conn.client_addr_port_=43866, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221735156, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221735156, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-000636654429FFDE-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82988, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418557991255433, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:68, fixup_index:-1, fixup_ash_buffer:{this:0x2b09651fcc38, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:58:30.554578] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDE-0-0] [lt=66][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:58:30.554664] INFO [SERVER] process (obmp_connect.cpp:504) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDE-0-0] [lt=19] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221735156, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:58:30.555456] INFO [SERVER] process (obmp_init_db.cpp:44) [83080][T1_L0_G0][T1][YB42AC1087A6-00063665502A138B-0-0] [lt=23] init db(db_name=oceanbase) [2025-05-31 02:58:30.559591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.562559] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:30.562587] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] try gc retain ctx [2025-05-31 02:58:30.562595] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-31 02:58:30.562630] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1748678310455421000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678300474165001, v:0}, keep_alive_scn_:{val:1748678305430518000, v:0}, update_ts_:1748678310559465}) [2025-05-31 02:58:30.563298] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678320364653000, latest_log_ts:{val:1748678303714352001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:30.570781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.579136] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:30.582035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.589035] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A63-0-0] [lt=31] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:30.590173] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.591042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.591083] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.591111] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.591136] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.592318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.592639] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.592904] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:30.602789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.614644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.624916] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.636431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.648134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.656751] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8BC-0-0] [lt=25] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.676449251174926758e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.678656597848224465e+03) [2025-05-31 02:58:30.657298] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=80, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:30.657344] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=4415, avg_group_log_size=441) [2025-05-31 02:58:30.657365] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4415) [2025-05-31 02:58:30.657380] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:30.661283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.662690] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2910, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:30.663101] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=408] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5004) [2025-05-31 02:58:30.663127] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=814, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:30.666257] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=48, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:30.666582] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11960][T1_L0_G1][T1][YB42AC1087A7-00063665517F53D1-0-0] [lt=34] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=114, avg_log_submit_to_flush_time=4151, avg_log_submit_to_slide_time=5283) [2025-05-31 02:58:30.667139] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4415, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:30.671648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.682679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.692950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.703684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.709171] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=58] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:30.709212] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-31 02:58:30.709223] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:30.709239] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:30.709254] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525358891181, flush_io_finished_ret_:0, flush_io_finished_round_:19878, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:30.714115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=168][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.716588] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4031895, max_lsn:2137102469, max_scn:{val:1748678310654256000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137102469}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095159429}}, last_submit_lsn:{lsn:2137101880}, last_submit_end_lsn:{lsn:2137102469}, last_submit_log_id:4031895, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137101880}, max_flushed_end_lsn:{lsn:2137102469}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137102469}, last_slide_log_id:4031895, last_slide_scn:{val:1748678310654256000, v:0}, last_slide_lsn:{lsn:2137101880}, last_slide_end_lsn:{lsn:2137102469}, last_slide_log_pid:11, last_slide_log_accum_checksum:2214974411, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4031896, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137102469}, readable_log_tail:{lsn:2137102469}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10250107, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:58:30.716943] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=344] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678310666699, last_flushed_end_lsn:{lsn:2137102469}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678310666242, last_flushed_end_lsn:{lsn:2137102469}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678310666520, last_flushed_end_lsn:{lsn:2137102469}}]) [2025-05-31 02:58:30.716987] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=37] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4031896, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678310666699, last_flushed_end_lsn:{lsn:2137102469}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678310666242, last_flushed_end_lsn:{lsn:2137102469}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678310666520, last_flushed_end_lsn:{lsn:2137102469}}]) [2025-05-31 02:58:30.725357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.735169] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4467-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:30.735716] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:30.735750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.749043] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:30.749131] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44670, [delay/us]:prepare:13, schedule:115, submit:33, rt:2313, total:2475) [2025-05-31 02:58:30.749152] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:30.749261] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=92] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44669, [delay/us]: prepare:13, schedule:115, submit:33, device:2313, total:2475) [2025-05-31 02:58:30.749276] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:30.749305] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4060, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:410, total:425) [2025-05-31 02:58:30.749330] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:30.749462] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=123] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4060, [delay/us]: prepare:0, schedule:0, submit:600000000, device:410, total:425) [2025-05-31 02:58:30.749537] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:30.749557] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:30.749566] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:30.749575] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:30.749643] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:30.749657] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=220kB/s, net_out=115kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:30.749723] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399406][2nd=2361]}) [2025-05-31 02:58:30.750420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.761598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.771996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.781017] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:30.782210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.794995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.806382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.807193] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=47] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137102591}, readable_log_tail:{lsn:2137102591}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10249985, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:30.807295] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=101] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:58:30.807328] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:58:30.817486] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA56-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:30.818172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.825406] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1748678310825401, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000493, add_bytes: 0 [2025-05-31 02:58:30.829230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.832941] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:58:30.832978] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=26] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:58:30.833000] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:30.842461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.843352] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:30.843375] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678310340554], gts:1748678310372724000, latest_srr:[mts=1748678310843329]}) [2025-05-31 02:58:30.845274] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC10870A-000636671448C58D-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1748678310835672000, start_id=1748678310835672001, end_id=1748678310835672002) [2025-05-31 02:58:30.855096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.866275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.879382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.897344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.897665] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=25] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:30.903419] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1748678310903415, bytes: 7125895612, bw: 0.224574 MB/s, add_ts: 1006463, add_bytes: 237005 [2025-05-31 02:58:30.904827] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F7-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525360086996, cache_obj->added_lc()=false, cache_obj->get_object_id()=53572, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:30.907722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.918805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.920376] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=26] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:58:30.928978] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:30.929401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.931208] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=105] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:30.936046] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:30.939639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.946989] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=142] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:58:30.951114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.957244] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B97-0-0] [lt=24] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678310985819, keepalive_ts:-1}) [2025-05-31 02:58:30.957283] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B97-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678310985819, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:30.961374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.971680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.981712] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=111] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:30.981915] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:30.983147] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=408941, clean_num=31457) [2025-05-31 02:58:30.994183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.004949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.013553] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=41] ====== check clog disk timer task ====== [2025-05-31 02:58:31.013609] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=46] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:31.017928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.028088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.035581] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=56] current monitor number(seq_=-1) [2025-05-31 02:58:31.035676] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=24] gc stale ls task succ [2025-05-31 02:58:31.038392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.039157] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:31.049133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.049595] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=88] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:31.062746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.070244] INFO [LIB] stat (utility.h:1211) [13069][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F9B-0-0] [lt=39] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:31.070291] INFO [LIB] stat (utility.h:1211) [13069][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F9B-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5627, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:31.073210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.075043] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678310835672000, v:0}, new_max_version={val:1748678310835672000, v:0}, last_min_version={val:1748678310654256000, v:0}, last_max_version={val:1748678310654256000, v:0}, total_time_us=9023, delta_us=239367) [2025-05-31 02:58:31.075086] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678310835672000, v:0}, min_version={val:1748678310835672000, v:0}, max_version={val:1748678310835672000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:31.085314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=205][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.097526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.108238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.114081] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AD8-0-0] [lt=60] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:31.118457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.129259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.136264] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:31.139977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.151815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.159320] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=76] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:31.159359] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=28] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:31.159375] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1043) [2025-05-31 02:58:31.162928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.173061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.184001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.186294] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:31.196179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.206811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.215167] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:31.217971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.220636] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=74] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:31.222640] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20] === Run print tenant memory usage task === [2025-05-31 02:58:31.222686] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 121,196,544 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 649,641,984 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,311,219,145 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 53,019,020 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 893,775,872 max_mem_memstore_can_get_now= 265,095,126 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:58:31.222736] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=45] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:31.223122] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,073,150,976 total_hold= 3,323,985,920 used= 1,576,685,568 freelists_hold= 1,496,465,408 total_maps= 2,017 total_unmaps= 805 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 121,196,544 virtual_memory_used= 4,344,737,792 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,002,438,656 count= 478 pushes= 14,512,214 pops= 14,511,736 maps= 1,822 unmaps= 761 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,930 pops= 22,926 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:58:31.223195] INFO print (ob_malloc_time_monitor.cpp:37) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:58:31.224702] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=48] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:31.224756] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=52] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:31.226535] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:31.226629] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=44] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:31.226647] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=16] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:31.226673] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:31.227259] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=582] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:31.227303] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=33] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:31.227313] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=8] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:31.227334] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:31.227353] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:31.228252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.229280] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=5] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:31.229330] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=46] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:31.229343] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:31.229378] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:31.229398] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=20] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:31.229407] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:31.229442] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=31] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:31.229462] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD5-0-0] [lt=18] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:31.238783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.249106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.256341] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=84] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:31.262067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.262124] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:31.264909] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [13069][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=11] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:31.269029] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0F9D-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3990) [2025-05-31 02:58:31.272615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.274251] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:495286 disk:48K/11:495286 t1m2:48K/11:495286 t1m2g0:48K/11:495286 |q0: 0 |q1: 0 [2025-05-31 02:58:31.279783] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:31.284302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.294984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.306808] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37F9-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525360488975, cache_obj->added_lc()=false, cache_obj->get_object_id()=53573, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.307295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.317532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.328010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.336683] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:31.338520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.344054] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:31.350437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.358639] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.358688] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.359334] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.359403] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.363461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.374166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.384553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.389198] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:31.397566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.409675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.420031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.429405] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:31.432045] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:31.433302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.443771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.454011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.464987] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=183][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.467826] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff8971d0, session_id:3221735156, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678310555977, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:822, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:31.467966] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=127] kill and revert session(conn.sessid_=3221735156, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:58:31.467983] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff896450, session_id:3221735158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678310537987, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:31.468003] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221735158, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:58:31.468041] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] sock ref clean, do destroy(*s={this:0x2b08ff8971d0, session_id:3221735156, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678310555977, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:822, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:31.468254] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=165] unlock session of tenant(conn.sessid_=3221735156, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:58:31.468271] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221735156, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-00063665502A138D-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:885018, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:885019, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:885020, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:885020, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:885020, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:885020, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:885020, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:16}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:885020, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:17}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:885020, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:18}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:58:31.468383] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=106] sock ref clean, do destroy(*s={this:0x2b08ff896450, session_id:3221735158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678310537987, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:31.468457] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221735158, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:58:31.468469] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221735158, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-00063665502A138A-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:1539150, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:1539151, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:1539152, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1539152, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1539152, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1539152, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1539152, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:1539152, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:1539152, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:58:31.470694] INFO [SERVER] run (obmp_disconnect.cpp:78) [83099][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=47] free session successfully(sessid=3221735156) [2025-05-31 02:58:31.470724] INFO [SERVER] run (obmp_disconnect.cpp:85) [83099][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=29] mark session id unused(sessid=3221735156) [2025-05-31 02:58:31.470793] INFO [SERVER] run (obmp_disconnect.cpp:78) [83099][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=8] free session successfully(sessid=3221735158) [2025-05-31 02:58:31.470804] INFO [SERVER] run (obmp_disconnect.cpp:85) [83099][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] mark session id unused(sessid=3221735158) [2025-05-31 02:58:31.475565] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [88871][T1_L0_G19][T1][YB42AC10871E-00063665511850D5-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:58:31.475608] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [88871][T1_L0_G19][T1][YB42AC10871E-00063665511850D5-0-0] [lt=34] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:58:31.476402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.486993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.499210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.510409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.526527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.537052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.537575] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:31.549369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.566538] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83090][T1_L0_G0][T1][YB42AC10870A-000636671698E3C8-0-0] [lt=15][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525360748695, cache_obj->added_lc()=false, cache_obj->get_object_id()=53574, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.572079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.586322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.589635] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:31.589943] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:31.589983] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.590021] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.590045] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.590045] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.591042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.591097] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.591127] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.591723] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.591776] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.592257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:31.596494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.604459] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] start to print blacklist info [2025-05-31 02:58:31.605071] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=31] ls blacklist refresh finish(cost_time=37726) [2025-05-31 02:58:31.608610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=168][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.627471] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:31.627720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.629402] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81742][T1_TimerWK1_ReqMemEvict][T1][YB42AC1087A6-0006366543FA5F2A-0-0] [lt=33] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=32) [2025-05-31 02:58:31.629835] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678311628212000, start_id=1748678311628212000, end_id=1748678311628212001) [2025-05-31 02:58:31.638497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:31.642282] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [14078][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:31.645076] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:31.655390] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=56861723, this={dio_aligned_buf:{buf_write_offset:1599, buf_padding_size:2497, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:389468, truncate_used_ts:132059}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=56860672, buf_len=548, total_write_size=43947, total_write_size_after_dio=475136, ob_pwrite_used_ts=369191, count=105, each_pwrite_cost=3516) [2025-05-31 02:58:31.655461] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137105435}], scn_array=[cnt:1, {val:1748678311628212000, v:0}], curr_size=548, accum_size=3514, time_cost=17560) [2025-05-31 02:58:31.655601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.660915] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=11] ====== traversal_flush timer task ====== [2025-05-31 02:58:31.661413] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B1-0-0] [lt=26] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678311465691000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678320364653000, latest_log_ts:{val:1748678303714352001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:31.661660] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B1-0-0] [lt=239] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:31.676026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.677772] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:31.677833] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4103, avg_group_log_size=373) [2025-05-31 02:58:31.677905] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4103) [2025-05-31 02:58:31.677922] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:31.679617] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=103] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5350, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:31.679666] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4103) [2025-05-31 02:58:31.680061] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=383] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1014, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:31.680311] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=88, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:31.680461] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FA2-0-0] [lt=32] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=630, avg_log_gen_to_submit_time=720, avg_log_submit_to_flush_time=6947, avg_log_submit_to_slide_time=7754) [2025-05-31 02:58:31.680834] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4103, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:31.686513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.698709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.709216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.709457] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:31.709523] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-05-31 02:58:31.709537] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:31.709554] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:31.709569] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525359891468, flush_io_finished_ret_:0, flush_io_finished_round_:19879, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:31.719468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.729969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.736905] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:31.736945] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=34] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:31.736959] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:31.736970] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:31.737815] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4468-0-0] [lt=103][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:31.738225] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:31.740109] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:31.741045] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=932] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:31.741574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.745983] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=49] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:31.752277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.762767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.772782] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:31.773101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.773127] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=288] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48000, [delay/us]:prepare:13, schedule:527, submit:540, rt:3921, total:5004) [2025-05-31 02:58:31.773159] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:31.773347] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=170] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:47995, [delay/us]: prepare:13, schedule:527, submit:540, device:3921, total:5004) [2025-05-31 02:58:31.773418] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:31.773448] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:3999, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1367, total:1518) [2025-05-31 02:58:31.773490] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:31.773525] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:3999, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1367, total:1518) [2025-05-31 02:58:31.773545] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:31.773561] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:31.773570] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:31.773578] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:31.773586] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:31.773596] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=200kB/s, net_out=153kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:31.773620] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399407][2nd=2643]}) [2025-05-31 02:58:31.783373] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.790400] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:31.795813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.806147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.816338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.826616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.832128] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1748678311832123, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006722, add_bytes: 0 [2025-05-31 02:58:31.838784] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83073][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA57-0-0] [lt=11][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:31.839434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.852929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.864284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.874903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.878572] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8C6-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700601196289062500e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633935583265972582e+03) [2025-05-31 02:58:31.885654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.895965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.898046] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=39] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:31.905728] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=98] PNIO [ratelimit] time: 1748678311905724, bytes: 7126090060, bw: 0.185013 MB/s, add_ts: 1002309, add_bytes: 194448 [2025-05-31 02:58:31.906320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.915284] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37FB-0-0] [lt=88][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525361097452, cache_obj->added_lc()=false, cache_obj->get_object_id()=53575, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.918479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.928938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.932121] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=86] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:31.932959] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:31.939594] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=110096, replace_num=15728) [2025-05-31 02:58:31.939694] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=97] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:31.940181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.950989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.959363] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83101][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33AF-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525361141531, cache_obj->added_lc()=false, cache_obj->get_object_id()=53576, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:31.962805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.973208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.983986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:31.993578] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:31.994951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=151][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.008476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.012706] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205D-0-0] [lt=31] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:32.012761] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205D-0-0] [lt=43] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=56) [2025-05-31 02:58:32.025171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=568][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.035608] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.036235] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=49] current monitor number(seq_=-1) [2025-05-31 02:58:32.036306] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=21] gc stale ls task succ [2025-05-31 02:58:32.039436] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:32.046035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.049759] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:32.056554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.066811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.077173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.082094] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1748678312078306000, start_id=1748678312078306000, end_id=1748678312078306001) [2025-05-31 02:58:32.083758] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:32.083783] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8662, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:32.087411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.091734] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678311673768000, v:0}, new_max_version={val:1748678311673768000, v:0}, last_min_version={val:1748678311628212000, v:0}, last_max_version={val:1748678311628212000, v:0}, total_time_us=13029, delta_us=417963) [2025-05-31 02:58:32.091779] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678311673768000, v:0}, min_version={val:1748678311673768000, v:0}, max_version={val:1748678311673768000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:32.097695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.107923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.108954] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:32.120666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.131102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.140199] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:32.141304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.151684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.161953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.172271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.182493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.192983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=320][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.194161] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:32.203498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=149][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.213959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.224819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.235380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.245953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.250379] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=51] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:32.250421] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=42] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:32.250541] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1096-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:32.250592] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1096-0-0] [lt=46] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:32.256295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.266603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.276980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.287967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.288611] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11960][T1_L0_G1][T1][YB42AC1087A7-00063665517F53E3-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5314) [2025-05-31 02:58:32.290797] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=33] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:376318 disk:52K/12:376318 t1m2:52K/12:376318 t1m2g0:52K/12:376318 |q0: 0 |q1: 0 [2025-05-31 02:58:32.298397] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.308538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.320116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.330925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.340704] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:32.343889] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.346432] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=72] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678312346411], gts:1748678312346127000, latest_srr:[mts=1748678312346411]}) [2025-05-31 02:58:32.346486] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:32.356777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.359065] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.359112] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.359190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.359218] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.360492] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=57] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:32.369011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.380099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.381557] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1748678312280679000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137110038}, palf_committed_end_scn_:{val:1748678312280679000, v:0}, last_check_scn_:{val:1748678312280679000, v:0}, max_applied_cb_scn_:{val:1748678312280679000, v:0}}) [2025-05-31 02:58:32.381616] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137110038}, readable_log_tail:{lsn:2137110038}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10242538, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:32.393989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.394757] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:32.404259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=141][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.414666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.428491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=346][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.432376] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:32.433185] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:32.441669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.452009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.462359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=149][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.473735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.474483] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C2-0-0] [lt=17] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:32.486090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.493793] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:32.498431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=146][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.509842] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.520508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=227][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.533606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=191][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.541508] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:32.541989] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=43] LogLoopThread round_cost_time(us)(round_cost_time=22) [2025-05-31 02:58:32.543886] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.555161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.565628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.571464] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A563B-0-0] [lt=43] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:32.577016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.588225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.590949] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.591324] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.591358] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.591383] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.591456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.591485] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.593569] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:32.595554] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:32.598401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.606426] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=32] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:32.610553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.624105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.635191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.648203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.648397] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:32.648619] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [6894][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=26] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:32.658811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.670968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=438][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.681247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.686171] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=86, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:32.686237] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4765, avg_group_log_size=433) [2025-05-31 02:58:32.686261] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4766) [2025-05-31 02:58:32.686277] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:32.686544] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD35-0-0] [lt=19] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678312682065, keepalive_ts:-1}) [2025-05-31 02:58:32.686608] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD35-0-0] [lt=60] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678312682065, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:32.691792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.693820] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=115] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5621, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:32.693899] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4765) [2025-05-31 02:58:32.693947] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=741, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:32.694142] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=1317, avg_log_gen_to_submit_time=1427, avg_log_submit_to_flush_time=6540, avg_log_submit_to_slide_time=6688) [2025-05-31 02:58:32.694200] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=130, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:32.695272] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:32.702039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.713713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.724045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.736192] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:32.736211] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:32.737053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.737381] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] update_check_range(min_check_version=1980, max_check_version=1981, global_version=1982) [2025-05-31 03:58:32.739103] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [14429][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:32.739157] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 03:58:32.739255] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [14429][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AB-0-0] [lt=118] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 03:58:32.739289] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [14429][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AB-0-0] [lt=23] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:58:32.742298] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4469-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:32.747616] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:32.750291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.761222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.770458] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=193] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 151 276 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 52 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 18 6 3 0 1 20 2 1 0 500 0 88 252 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 3 2 0 500 16 1 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 33107 [2025-05-31 02:58:32.770621] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=130] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 893,775,872 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 451,256,320 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 109,051,904 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 12,603,392 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:32.771532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.771722] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=43] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 451,256,320 used= 340,215,856 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,236,544 used= 16,017,984 count= 30 avg_used= 533,932 block_cnt= 30 chunk_cnt= 18 mod=MysqlRequesReco [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,200,768 used= 6,324,224 count= 107 avg_used= 59,104 block_cnt= 107 chunk_cnt= 28 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,818,048 used= 2,380,480 count= 86 avg_used= 27,680 block_cnt= 86 chunk_cnt= 26 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 999,424 used= 490,928 count= 61 avg_used= 8,048 block_cnt= 61 chunk_cnt= 21 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 407,296 used= 390,784 count= 86 avg_used= 4,544 block_cnt= 86 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 286,528 used= 275,824 count= 5 avg_used= 55,164 block_cnt= 5 chunk_cnt= 5 mod=CommSysVarFac [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 101,344 used= 26,256 count= 372 avg_used= 70 block_cnt= 100 chunk_cnt= 27 mod=Coro [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,472 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=ResultSet [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 56,160 used= 43,680 count= 65 avg_used= 672 block_cnt= 60 chunk_cnt= 23 mod=[T]ObWarningBuf [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 48,672 used= 34,048 count= 76 avg_used= 448 block_cnt= 43 chunk_cnt= 11 mod=OMT_Worker [MEMORY] hold= 45,536 used= 4,848 count= 202 avg_used= 24 block_cnt= 75 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSession [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,880 used= 1,920 count= 5 avg_used= 384 block_cnt= 5 chunk_cnt= 5 mod=TableLockNode [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceIdMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=ContextsMap [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 340,215,856 used= 332,159,027 count= 32,603 avg_used= 10,187 mod=SUMMARY [2025-05-31 02:58:32.771986] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=125] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:58:32.772044] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=43] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:58:32.772100] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=39] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,574,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,511,808 used= 1,476,096 count= 186 avg_used= 7,936 block_cnt= 186 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,360,000 used= 1,240,000 count= 625 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,574,720 used= 35,241,449 count= 3,357 avg_used= 10,497 mod=SUMMARY [2025-05-31 02:58:32.772145] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:58:32.772184] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 109,051,904 used= 105,799,680 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 105,799,680 used= 105,602,880 count= 205 avg_used= 515,136 block_cnt= 205 chunk_cnt= 52 mod=CoStack [MEMORY] hold= 105,799,680 used= 105,602,880 count= 205 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:58:32.772229] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:58:32.772269] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:58:32.772314] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=35] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:32.772351] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:32.772388] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:58:32.772434] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 12,603,392 used= 8,945,280 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,754,624 used= 5,618,688 count= 708 avg_used= 7,936 block_cnt= 708 chunk_cnt= 4 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 8,945,280 used= 8,780,352 count= 712 avg_used= 12,331 mod=SUMMARY [2025-05-31 02:58:32.772475] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:32.772545] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=35] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 645,447,680 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:32.773018] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 180,822,048 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT start_service_time FROM __all_server WHERE svr_ip = ? AND svr_port = ?) [2025-05-31 02:58:32.789327] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:32.789385] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48364, [delay/us]:prepare:18, schedule:396, submit:295, rt:4776, total:5487) [2025-05-31 02:58:32.789405] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:32.789467] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48372, [delay/us]: prepare:18, schedule:396, submit:295, device:4776, total:5487) [2025-05-31 02:58:32.789481] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:32.789507] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4031, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:464, total:480) [2025-05-31 02:58:32.789533] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:32.789562] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4031, [delay/us]: prepare:0, schedule:0, submit:600000000, device:464, total:480) [2025-05-31 02:58:32.789580] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:32.789594] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:32.789603] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:32.789611] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:32.789620] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:32.789629] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=161kB/s, net_out=110kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:32.789651] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399408][2nd=2938]}) [2025-05-31 02:58:32.797272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.798012] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:32.808607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.821503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.832355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.842516] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678312842510, bytes: 0, bw: 0.000000 MB/s, add_ts: 1010387, add_bytes: 0 [2025-05-31 02:58:32.846993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.848256] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1748678312847919000, start_id=1748678312847919000, end_id=1748678312847919001) [2025-05-31 02:58:32.857634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.857760] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=7] ====== tenant freeze timer task ====== [2025-05-31 02:58:32.857806] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=37] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:32.857832] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:32.860933] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A7-00063665435E04BE-0-0] [lt=110] get number(ret=0, service_type_=0, range=1, base_id=1748678312860304000, start_id=1748678312860304000, end_id=1748678312860304001) [2025-05-31 02:58:32.868727] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.873294] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA58-0-0] [lt=84][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:32.873747] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [13661][T1_L0_G1][T1][YB42AC1087A7-00063665517F53E9-0-0] [lt=38] init point thread id with(&point=0x55c0908dcb00, idx_=1373, point=[thread id=13661, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=13661) [2025-05-31 02:58:32.874410] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137111459}], scn_array=[cnt:1, {val:1748678312860304000, v:0}], curr_size=548, accum_size=670, time_cost=13141) [2025-05-31 02:58:32.874586] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=7] [APPLY STAT CB APPEND COST TIME](cur_stat_count=40, stat_interval=5000000, avg (count/cost)=173, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:32.874609] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=23] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=40, stat_interval=5000000, avg (count/cost)=7127, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:32.874618] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=40, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:32.874625] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=40, stat_interval=5000000, avg (count/cost)=-960, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:32.874632] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=40, stat_interval=5000000, avg (count/cost)=6340, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:32.879792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.890444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.898771] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:32.901309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:32.904084] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [14466][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:32.904186] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:32.908687] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1748678312908683, bytes: 7126261794, bw: 0.163295 MB/s, add_ts: 1002959, add_bytes: 171734 [2025-05-31 02:58:32.915284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.925490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.926014] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37FD-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525362108182, cache_obj->added_lc()=false, cache_obj->get_object_id()=53577, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:32.933533] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=75] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=524288, config_mem_limit_=32212254, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=6239232, evict_high_mem_level=16106127, evict_low_mem_level=0) [2025-05-31 02:58:32.933572] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:32.934976] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:32.935976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.946336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.964114] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=71] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:32.964288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.972154] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=32] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:32.974537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.985098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.996688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:32.998791] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=58] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:33.000694] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=132] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=440398, clean_num=31457) [2025-05-31 02:58:33.007084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.014047] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-05-31 02:58:33.014083] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:33.019199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.033406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.038086] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2025-05-31 02:58:33.038162] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=33] gc stale ls task succ [2025-05-31 02:58:33.039928] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:33.044056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.049976] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:33.056919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.068565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.078977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.089537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.092782] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8D0-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.724600601196289062e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.590417701883752216e+03) [2025-05-31 02:58:33.098975] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FB2-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=19, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:33.099059] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FB2-0-0] [lt=81] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=8438, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:33.099418] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678312860304000, v:0}, new_max_version={val:1748678312860304000, v:0}, last_min_version={val:1748678312681037000, v:0}, last_max_version={val:1748678312681037000, v:0}, total_time_us=9031, delta_us=239111) [2025-05-31 02:58:33.099452] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678312860304000, v:0}, min_version={val:1748678312860304000, v:0}, max_version={val:1748678312860304000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:33.099859] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.110126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.122433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.132683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.143097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=141][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.153439] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.163775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.164358] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:33.174144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.184832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.196575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.201378] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:33.206796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.217354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.229320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.239589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.249816] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.255562] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=13] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:33.257773] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=166] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:33.258683] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=49] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678312108758000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:33.260709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.273316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.283850] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.298270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.301848] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:40K/9:118838 disk:40K/9:118838 t1m2:40K/9:118838 t1m2g0:40K/9:118838 |q0: 0 |q1: 0 [2025-05-31 02:58:33.303001] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FB4-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4102) [2025-05-31 02:58:33.310780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.323032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.330988] INFO [STORAGE.COMPACTION] schedule_all_tablets_minor (ob_tenant_tablet_scheduler.cpp:553) [81742][T1_TimerWK1_MergeLoop][T1][YB42AC1087A6-0006366543FA5F2F-0-0] [lt=12] start schedule all tablet minor merge(minor_ls_tablet_iter_={ls_idx:-1, ls_ids:[{id:1}], tablet_ids:{tablet_idx:0, tablet_cnt:0, array:[], is_inited:false}, schedule_tablet_cnt:0, max_batch_tablet_cnt:50000}) [2025-05-31 02:58:33.333284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.343883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.349258] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=57] ts response task statistics(alloc_count=485666, free_count=242832) [2025-05-31 02:58:33.349315] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:33.349433] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678312848330], gts:1748678312883235000, latest_srr:[mts=1748678313349334]}) [2025-05-31 02:58:33.354450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.361377] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.361470] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=57] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.361522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.361568] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.364690] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:33.366786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.377664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.389484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.399831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.401851] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:33.410319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.420656] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.431597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.434158] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:33.437967] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:33.441819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.445234] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B1C-0-0] [lt=47] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.452144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.462366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.472583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.474493] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:78) [81742][T1_TimerWK1_MergeLoop][T1][YB42AC1087A6-0006366543FA5F2F-0-0] [lt=91] MergeLoopTask(cost_ts=143746) [2025-05-31 02:58:33.484165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.496752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.500894] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A1D-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525362683065, cache_obj->added_lc()=false, cache_obj->get_object_id()=53578, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:33.507313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.517471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.527956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.538262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.549014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.559617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.565671] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=158] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:33.571189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.581554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.587177] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B1D-0-0] [lt=26] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.589948] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=345] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592348] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592436] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592671] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592788] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.592931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.593032] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:33.600154] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A64-0-0] [lt=104] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:33.601149] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:33.603106] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:33.603348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.615661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.622364] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B1E-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.626366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.636674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.647058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.657529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.668196] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=44] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:33.668257] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=45] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1026) [2025-05-31 02:58:33.668590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.680944] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B1F-0-0] [lt=33] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.681567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.693188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.703698] INFO [LIB] stat (utility.h:1211) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=87, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:33.703747] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3520, avg_group_log_size=352) [2025-05-31 02:58:33.703823] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=69] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3980) [2025-05-31 02:58:33.703911] INFO [LIB] stat (utility.h:1211) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=85] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:33.703927] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=12] submit log success(service_type_=0, last_id=1748678330364653000, limited_id=1748678330364653000) [2025-05-31 02:58:33.703938] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=11] submit log with lock success(service_type_=0, last_id=1748678330364653000, limited_id=1748678330364653000) [2025-05-31 02:58:33.705076] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.707484] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:100) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] is logging [2025-05-31 02:58:33.707525] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:48) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] submit log with lock(ret=-4023, service_type_=0, last_id=1748678330364653000, limited_id=1748678330364653000) [2025-05-31 02:58:33.710778] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5135, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:33.710810] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3520) [2025-05-31 02:58:33.710832] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=754, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:33.713401] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=79] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:33.713493] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-05-31 02:58:33.713508] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:33.713525] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:33.713540] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525361895505, flush_io_finished_ret_:0, flush_io_finished_round_:19881, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:33.714167] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:100) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=7] is logging [2025-05-31 02:58:33.714189] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:48) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672E5E1672-0-0] [lt=20] submit log with lock(ret=-4023, service_type_=0, last_id=1748678330364653000, limited_id=1748678330364653000) [2025-05-31 02:58:33.715143] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=135, avg_log_submit_to_flush_time=6358, avg_log_submit_to_slide_time=7314) [2025-05-31 02:58:33.715216] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=110, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:33.715299] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1748678330364653000, latest_scn={val:1748678313702670001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:58:33.715385] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=84] handle submit callback(service_type_=0, success=true, used_ts=11460, log_ts={val:1748678313702670001, v:0}, limited_id=1748678330364653000) [2025-05-31 02:58:33.715402] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=15] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:58:33.715412] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A37F6-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:58:33.715548] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3520, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:33.715779] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.721178] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [14078][T1_L0_G1][T1][YB42AC1087A7-00063665517F53F2-0-0] [lt=27] init point thread id with(&point=0x55c0908e3340, idx_=1790, point=[thread id=14078, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=14078) [2025-05-31 02:58:33.728235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.738536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.738632] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:58:33.745164] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446A-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:33.750452] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [88871][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:58:33.751332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.763962] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B20-0-0] [lt=54] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.764158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.769039] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:33.774377] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.784748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.795743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=181][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.801164] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:33.801235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48576, [delay/us]:prepare:13, schedule:458, submit:123, rt:4201, total:4797) [2025-05-31 02:58:33.801257] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:33.801402] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=128] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48572, [delay/us]: prepare:13, schedule:458, submit:123, device:4201, total:4797) [2025-05-31 02:58:33.801464] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:33.801648] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4047, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2779, total:2841) [2025-05-31 02:58:33.801759] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:33.801811] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4046, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2779, total:2841) [2025-05-31 02:58:33.801833] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:33.802025] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:33.802051] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:33.802068] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:33.802084] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:33.802099] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=179kB/s, net_out=98kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:33.802133] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] successfully alloc block(macro_id={[ver=1,mode=0,seq=399409][2nd=2092]}) [2025-05-31 02:58:33.804733] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:33.805919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.816552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.827849] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.839384] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.848610] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=782] PNIO [ratelimit] time: 1748678313848604, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006094, add_bytes: 0 [2025-05-31 02:58:33.849881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.850741] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678313349725], gts:1748678313382544000, latest_srr:[mts=1748678313850713]}) [2025-05-31 02:58:33.861172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.871521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.884637] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.892787] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B21-0-0] [lt=37] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.894938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.898035] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA59-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:33.899205] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=281] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:33.905269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.905377] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:33.905968] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [13661][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:33.911833] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1748678313911830, bytes: 7126430718, bw: 0.160593 MB/s, add_ts: 1003147, add_bytes: 168924 [2025-05-31 02:58:33.915851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.927860] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B22-0-0] [lt=43] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:33.928303] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=27] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:58:33.928748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.934393] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:33.937351] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A37FF-0-0] [lt=158][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525363119520, cache_obj->added_lc()=false, cache_obj->get_object_id()=53579, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:33.940796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.941285] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:33.949187] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=22] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:33.954402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.967326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.973058] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:33.979451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:33.990488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.001173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.005600] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:34.011562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.017121] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205E-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:34.017168] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205E-0-0] [lt=35] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=49) [2025-05-31 02:58:34.023760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.038332] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=35] current monitor number(seq_=-1) [2025-05-31 02:58:34.038460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.038492] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=18] gc stale ls task succ [2025-05-31 02:58:34.040321] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:34.048965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.053223] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:34.059247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.069708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.080411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.093653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.104690] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.111716] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8DB-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700601196289062500e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633935583265972582e+03) [2025-05-31 02:58:34.115309] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FBB-0-0] [lt=173] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=16, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:34.115366] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FBB-0-0] [lt=56] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=7692, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:34.118000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.120081] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678313849346000, v:0}, new_max_version={val:1748678313849346000, v:0}, last_min_version={val:1748678313702670001, v:0}, last_max_version={val:1748678313702670001, v:0}, total_time_us=10551, delta_us=270732) [2025-05-31 02:58:34.120140] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678313849346000, v:0}, min_version={val:1748678313849346000, v:0}, max_version={val:1748678313849346000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:34.126030] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=35] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:34.128294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.136964] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3ADA-0-0] [lt=43] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:34.137745] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1748678314137738, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:58:34.137797] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=50] snapshot_traffic_info once [2025-05-31 02:58:34.138688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.149038] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.159252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.170607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.177114] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=125824, replace_num=15728) [2025-05-31 02:58:34.177166] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:34.181277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.191832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.202260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.206217] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:34.209992] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678314111223000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678314209931}, version_delta=98765) [2025-05-31 02:58:34.215728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.226548] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.237722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.247975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.252160] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=33] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:34.252187] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:34.254182] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1097-0-0] [lt=15] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:34.254215] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1097-0-0] [lt=28] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:34.258403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=156][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.264516] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=42] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:34.264642] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4149-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1748678314264155000, start_id=1748678314264155000, end_id=1748678314264155001) [2025-05-31 02:58:34.268970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=270][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.279252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.289798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.300504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.313696] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3505) [2025-05-31 02:58:34.316385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.316629] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=40] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:788869 disk:48K/11:788869 t1m2:48K/11:788869 t1m2g0:48K/11:788869 |q0: 0 |q1: 0 [2025-05-31 02:58:34.321990] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83082][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3801-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525363504158, cache_obj->added_lc()=false, cache_obj->get_object_id()=53580, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:34.327023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.332378] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3803-0-0] [lt=43][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:34.336275] WDIAG get_refactored (ob_hashtable.h:1120) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3805-0-0] [lt=8][errcode=-4201] hashtable not init [2025-05-31 02:58:34.338026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.348315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.351273] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=43] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:34.351382] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=68] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=2, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:34.359814] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.360007] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=59] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.360051] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.360050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.360080] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.370389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.375332] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=44] blacklist_loop exec finished(cost_time=67, is_enabled=true, send_cnt=0) [2025-05-31 02:58:34.377329] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:34.381271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.391484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.401768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.406943] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:34.413114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.423792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=162][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.436242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.436667] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=72] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:34.441565] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=78] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:34.446721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.457032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.470359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.474371] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F8E3F05-0-0] [lt=120] get number(ret=0, service_type_=0, range=1, base_id=1748678314472181000, start_id=1748678314472181000, end_id=1748678314472181001) [2025-05-31 02:58:34.484781] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C3-0-0] [lt=75] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:34.486563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.498836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.501316] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B23-0-0] [lt=127] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:34.510503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.520765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.531898] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.543008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.543216] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B24-0-0] [lt=30] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:34.545110] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:670) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=26] failed to get result(ret=-4008) [2025-05-31 02:58:34.545250] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:515) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=19] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748678246784954, start_date_:1748169746138009, last_date_:1748678246141150, this_date_:0, next_date_:1748679146138009, end_date_:64060560000000000, total_:239131641, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748613666964441, start_date_:1748613600000000, last_date_:1748613600006788, this_date_:0, next_date_:1749218400000000, end_date_:64060560000000000, total_:66957653, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748314024043185, start_date_:1748268000000000, last_date_:0, this_date_:0, next_date_:1748872800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748600847610427, start_date_:1748255246137533, last_date_:1748600846140058, this_date_:0, next_date_:1748687246137533, end_date_:64060560000000000, total_:3757845, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748168846136670, start_date_:1748700000000000, last_date_:0, this_date_:0, next_date_:1748700000000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748624400066065, start_date_:1748192400000000, last_date_:1748624400003705, this_date_:0, next_date_:1748710800000000, end_date_:64060560000000000, total_:347108, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:4, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748181773502269, start_date_:1748181600000000, last_date_:1748181600002317, this_date_:0, next_date_:1748786400000000, end_date_:64060560000000000, total_:173499952, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748527281005724, start_date_:1748527200000000, last_date_:1748527200011176, this_date_:0, next_date_:1749132000000000, end_date_:64060560000000000, total_:80994548, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748437312134789, start_date_:1748354400000000, last_date_:0, this_date_:0, next_date_:1748959200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748440876223997, start_date_:1748440800000000, last_date_:1748440800003333, this_date_:0, next_date_:1749045600000000, end_date_:64060560000000000, total_:76220664, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}]) [2025-05-31 02:58:34.545385] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:504) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=133] check all tenants(ret=0) [2025-05-31 02:58:34.553267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.563512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.573796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.578051] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:34.584179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.590932] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.591530] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.591588] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.591618] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.591643] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.591668] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.592271] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:34.594953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.597807] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:34.597907] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:34.597977] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:34.598037] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:34.605282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.607537] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:34.608622] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=51] ls blacklist refresh finish(cost_time=40954) [2025-05-31 02:58:34.611826] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678314335778000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=276034, in_cluster_service=true, cluster_version={val:1748678314309499000, v:0}, min_cluster_version={val:1748678314309499000, v:0}, max_cluster_version={val:1748678314309499000, v:0}, get_cluster_version_err=0, cluster_version_delta=302313, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=194993, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=305557054, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:34.615756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.626261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.640418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.650782] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.663435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=216][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.666208] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83104][T1_L0_G0][T1][YB42AC10871E-000636654548A523-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678314662815000, start_id=1748678314662815000, end_id=1748678314662815001) [2025-05-31 02:58:34.675913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.687991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.698283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.708958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.715264] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=100] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=119, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:34.715322] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=6176, avg_group_log_size=475) [2025-05-31 02:58:34.715344] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5715) [2025-05-31 02:58:34.715359] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:34.718501] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3183, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:34.718568] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=65] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=6176) [2025-05-31 02:58:34.718591] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1052, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:34.718672] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=104] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:34.719217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.721015] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FC4-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=1013, avg_log_gen_to_submit_time=1149, avg_log_submit_to_flush_time=4288, avg_log_submit_to_slide_time=6706) [2025-05-31 02:58:34.721133] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=6176, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 03:58:34.721442] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [14998][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:34.721576] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:34.723158] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:34.733418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.743719] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.748045] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446B-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:34.754055] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.763259] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=34][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:34.764424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.774656] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.779021] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:34.785064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.796178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=175][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.807041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.807796] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:34.807944] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, norm_iops:3, bw:56966, [delay/us]:prepare:13, schedule:421, submit:75, rt:2515, total:3025) [2025-05-31 02:58:34.807970] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:34.808086] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=99] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56963, [delay/us]: prepare:13, schedule:421, submit:75, device:2515, total:3025) [2025-05-31 02:58:34.808102] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:34.808130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4069, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1632, total:1649) [2025-05-31 02:58:34.808157] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:34.808189] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4070, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1632, total:1649) [2025-05-31 02:58:34.808207] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:34.808222] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:34.808231] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:34.808239] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:34.808248] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:34.808258] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=161kB/s, net_out=146kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:34.808280] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399410][2nd=2374]}) [2025-05-31 02:58:34.808406] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:34.820411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.830948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.843609] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.853919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.854236] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=55] PNIO [ratelimit] time: 1748678314854231, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005627, add_bytes: 0 [2025-05-31 02:58:34.863427] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=31] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:34.864131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.866731] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=77] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19341254656, fast_delete_log_mode=false) [2025-05-31 02:58:34.874345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.884691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.892126] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:58:34.892211] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=73] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:58:34.892237] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:34.895349] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.899362] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=31] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:34.907106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.915700] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=76] PNIO [ratelimit] time: 1748678314915687, bytes: 7126605530, bw: 0.166073 MB/s, add_ts: 1003857, add_bytes: 174812 [2025-05-31 02:58:34.917658] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678314711984000, v:0}, base_version={val:1748678314472181000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678314711984000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678314917635, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678314711984000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678314894769, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678314711984000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678314897662, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678309764155000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678314901304, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678309935670000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678314896616, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678310150188000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678314880918, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:58:34.918825] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA5A-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:34.919309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.928170] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=60] success to update min reserved snapshot(reserved_snapshot=1748676512108758000, duration=1800, snapshot_gc_ts=1748678312108758000) [2025-05-31 02:58:34.932233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.933623] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=30] update reserved snapshot finished(cost_ts=5458, reserved_snapshot=1748676512108758000) [2025-05-31 02:58:34.938401] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:34.942606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.945655] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=82] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:34.946572] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81426][pnio1][T0][YB42AC10870A-000636671AB94600-0-0] [lt=39][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:3110571160, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678314970987, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1378708, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678314955480, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678314946567, enqueue_timestamp_:0, request_arrival_time_:1748678314946567, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:34.946678] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81426][pnio1][T0][YB42AC10870A-000636671AB94600-0-0] [lt=78][errcode=-5150] deliver rpc request fail(&req=0x2b0959610098, ret=-5150) [2025-05-31 02:58:34.953267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.959234] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83074][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3807-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525364141403, cache_obj->added_lc()=false, cache_obj->get_object_id()=53581, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:34.965232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=246][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.968519] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [8165][T1_L0_G1][T1][YB42AC10871E-0006366546E85B98-0-0] [lt=41] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678314999637, keepalive_ts:-1}) [2025-05-31 02:58:34.968579] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [8165][T1_L0_G1][T1][YB42AC10871E-0006366546E85B98-0-0] [lt=57] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678314999637, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:34.978199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:34.979238] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:34.988777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.000260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.009465] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:35.011020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.011417] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=471855, clean_num=31457) [2025-05-31 02:58:35.016221] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-05-31 02:58:35.016268] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=39] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:35.021371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.031610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.038760] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=54] current monitor number(seq_=-1) [2025-05-31 02:58:35.038841] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:58:35.043043] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:35.045137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.053682] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=335] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:35.057953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.068188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.081115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.084578] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81744][T1_TimerWK3_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=50] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:35.091367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.102068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.113776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.122060] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8E6-0-0] [lt=76] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700601196289062500e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633935583265972582e+03) [2025-05-31 02:58:35.124100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.124955] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137122404}], scn_array=[cnt:1, {val:1748678314945640001, v:0}], curr_size=122, accum_size=1493, time_cost=10218) [2025-05-31 02:58:35.125173] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=21, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:35.125195] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=8262, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:35.131486] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678314916509001, v:0}, new_max_version={val:1748678314916509001, v:0}, last_min_version={val:1748678314711984000, v:0}, last_max_version={val:1748678314711984000, v:0}, total_time_us=12045, delta_us=214973) [2025-05-31 02:58:35.131588] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678314916509001, v:0}, min_version={val:1748678314916509001, v:0}, max_version={val:1748678314916509001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:35.135657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.146632] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA63-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525364328801, cache_obj->added_lc()=false, cache_obj->get_object_id()=53582, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:35.147696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.158016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.168561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.179607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.179977] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=73] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:35.189894] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.200206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.212673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.212679] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:35.223060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.236246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.246924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.257146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.267799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.278191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.288496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.298794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.309161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.318274] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=6181) [2025-05-31 02:58:35.318691] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3809-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525364500860, cache_obj->added_lc()=false, cache_obj->get_object_id()=53583, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:35.319428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.326122] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=64] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:60K/14:452901 disk:60K/14:452901 t1m2:60K/14:452901 t1m2g0:60K/14:452901 |q0: 0 |q1: 0 [2025-05-31 02:58:35.330295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.341135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.351730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.355549] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=671, get_gts_with_stc_cnt=36, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=34, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:35.355587] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678315355508], gts:1748678315355165000, latest_srr:[mts=1748678315355508]}) [2025-05-31 02:58:35.355607] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:35.359160] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=214] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.359984] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.360042] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.360073] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.360737] INFO pn_ratelimit (group.c:653) [81065][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=22] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:35.362125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.372472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.380721] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:35.382677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.393566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.404279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.406054] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC108714-000636671E6D511B-0-0] [lt=50][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525364588222, cache_obj->added_lc()=false, cache_obj->get_object_id()=53584, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:35.415956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.416798] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:35.427132] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.438084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.441160] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-05-31 02:58:35.445834] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:35.448418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.458891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.470451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.483988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.494244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.505776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.516533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.516730] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=34] ====== checkpoint timer task ====== [2025-05-31 02:58:35.517758] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:35.517889] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=129] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:35.518263] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=65] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:35.518357] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=89] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:35.518418] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:35.518433] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1748678313702670001, v:0}) [2025-05-31 02:58:35.518442] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:35.518449] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:35.518464] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:35.518485] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:35.518499] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:35.518519] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:35.518542] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:35.518552] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:35.529261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.539701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.550210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.562480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.572806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.582373] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:35.583092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.590918] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.591957] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.591998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.592051] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=83] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.592815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.592915] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.592962] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:35.593376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.603715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.616686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.617787] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=107] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:35.620073] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=81] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:35.620142] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=65] try gc retain ctx [2025-05-31 02:58:35.620153] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] try to retry start_working [2025-05-31 02:58:35.620200] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678315524251000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678305430518000, v:0}, keep_alive_scn_:{val:1748678310455421001, v:0}, update_ts_:1748678315618916}) [2025-05-31 02:58:35.620847] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:35.627247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.637506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.650196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.660543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.670803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.681652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.691899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.702253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.712600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.715718] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=97] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:35.715778] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51] temporary file module use no blocks [2025-05-31 02:58:35.715792] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:35.715808] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:35.715824] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525363897598, flush_io_finished_ret_:0, flush_io_finished_round_:19883, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:35.721066] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=191] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=88, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:35.721125] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=3739, avg_group_log_size=311) [2025-05-31 02:58:35.721148] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4206) [2025-05-31 02:58:35.721164] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:35.723134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.723261] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:35.723319] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [12787][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=45] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:35.726683] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4548, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:35.726711] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3739) [2025-05-31 02:58:35.726733] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=980, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:35.726795] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=538, avg_log_gen_to_submit_time=644, avg_log_submit_to_flush_time=5813, avg_log_submit_to_slide_time=6751) [2025-05-31 02:58:35.726904] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:35.728380] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3739, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:35.735668] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:35.736234] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [15355][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 03:58:35.736298] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [15355][T1_L0_G1][T1][YB42AC1087A7-00063665517F540A-0-0] [lt=31] init point thread id with(&point=0x55c0908f7280, idx_=3067, point=[thread id=15355, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=15355) [2025-05-31 02:58:35.738747] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:35.751190] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446C-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:35.751792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.762095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.772490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.782595] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:35.783202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.794270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.796436] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=29] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:58:35.804533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.814823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.818520] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:35.820920] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:35.821073] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56596, [delay/us]:prepare:15, schedule:505, submit:396, rt:3200, total:4117) [2025-05-31 02:58:35.821096] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:35.821153] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56604, [delay/us]: prepare:15, schedule:505, submit:396, device:3200, total:4117) [2025-05-31 02:58:35.821167] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:35.821235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4043, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:507, total:523) [2025-05-31 02:58:35.821266] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:35.821299] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4043, [delay/us]: prepare:0, schedule:0, submit:600000000, device:507, total:523) [2025-05-31 02:58:35.821318] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:35.821333] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:35.821341] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:35.821350] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:35.821358] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:35.821367] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=158kB/s, net_out=107kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:35.821388] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399411][2nd=2656]}) [2025-05-31 02:58:35.825299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.831428] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:35.835681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.846294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.851128] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC108714-000636671D6D3388-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1748678315850316000, start_id=1748678315850316000, end_id=1748678315850316001) [2025-05-31 02:58:35.858681] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:35.859023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.863398] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1748678315863395, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009164, add_bytes: 0 [2025-05-31 02:58:35.869265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.879498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.889845] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.900931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=256][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.901235] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=49] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:35.914280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=148][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.923072] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=92] PNIO [ratelimit] time: 1748678315923067, bytes: 7126768831, bw: 0.154595 MB/s, add_ts: 1007380, add_bytes: 163301 [2025-05-31 02:58:35.924827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.935184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.944628] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=33] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:35.945510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.948513] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=146] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:35.956989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=141][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.967370] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.972011] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A380B-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525365154179, cache_obj->added_lc()=false, cache_obj->get_object_id()=53585, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:35.977596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.985418] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:35.987884] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:35.994201] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA5B-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:36.000121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.006536] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:58:36.006673] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=48] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:58:36.011282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.020128] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:36.021965] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205F-0-0] [lt=217] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:36.022002] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A205F-0-0] [lt=28] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=39) [2025-05-31 02:58:36.022122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.032378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.039248] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=90] gc stale ls task succ [2025-05-31 02:58:36.043191] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:36.045881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.054073] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=49] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:36.056170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.056403] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A7-00063665469F7AFD-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1748678316055720000, start_id=1748678316055720000, end_id=1748678316055720001) [2025-05-31 02:58:36.066462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.076801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=136][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.087140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.097673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.108032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.117206] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:36.118240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.129093] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [14466][T1_L0_G1][T1][YB42AC1087A7-00063665517F540E-0-0] [lt=179] init point thread id with(&point=0x55c0908e9440, idx_=2178, point=[thread id=14466, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=14466) [2025-05-31 02:58:36.129216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.130851] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:36.130989] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=136] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5921, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:36.139031] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF8F3-0-0] [lt=48] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.724600601196289062e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.590417701883752216e+03) [2025-05-31 02:58:36.139410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.139623] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=123] get number(ret=0, service_type_=0, range=1, base_id=1748678316139084000, start_id=1748678316139084000, end_id=1748678316139084001) [2025-05-31 02:58:36.145933] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678315855556000, v:0}, new_max_version={val:1748678315855556000, v:0}, last_min_version={val:1748678315524251002, v:0}, last_max_version={val:1748678315524251002, v:0}, total_time_us=9301, delta_us=290373) [2025-05-31 02:58:36.145992] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678315855556000, v:0}, min_version={val:1748678315855556000, v:0}, max_version={val:1748678315855556000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:36.149734] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.159444] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:36.159519] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=67] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:36.159552] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=17] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:36.159568] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=11] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1009) [2025-05-31 02:58:36.160666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.171126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.181783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.187414] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:36.192100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.204334] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.217702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.221024] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:36.228660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.242049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.247253] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:36.251970] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=74] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:36.256743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.257359] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=157] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:36.258043] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=45] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:36.260633] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=39] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:36.260722] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=87] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:36.262969] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:36.263040] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:36.263054] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:36.263080] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:36.263936] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=850] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:36.263977] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=31] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:36.263989] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:36.264011] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:36.264030] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=18] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:36.268461] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:36.268458] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=58] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:36.268484] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=22] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:36.268494] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:36.268533] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:36.268601] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=67] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:36.268612] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=10] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:36.268625] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=10] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:36.268642] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD6-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:36.271296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=291][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.275337] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=29] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:36.275686] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=47] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:36.275714] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=28] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:36.275792] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1098-0-0] [lt=13] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:36.275814] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1098-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:36.276002] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=44] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678312108758000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:36.281497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.291741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.305314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.321111] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.330136] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:262475 disk:44K/10:262475 t1m2:44K/10:262475 t1m2g0:44K/10:262475 |q0: 0 |q1: 0 [2025-05-31 02:58:36.331829] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FD6-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3668) [2025-05-31 02:58:36.333081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.344353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.356090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.359357] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.361473] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=32] ts response task statistics(alloc_count=485738, free_count=242868) [2025-05-31 02:58:36.361531] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:36.361643] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678315858222], gts:1748678315892979000, latest_srr:[mts=1748678316361550]}) [2025-05-31 02:58:36.362270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.362328] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.362358] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.366665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.376990] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.387708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.389365] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=141552, replace_num=15728) [2025-05-31 02:58:36.389408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:36.398005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.408257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.420270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.421727] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:36.430630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.443572] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.445341] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:36.453244] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=104] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:36.454995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.467889] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.479073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.489494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.498257] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C4-0-0] [lt=13] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:36.500844] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.511242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.524831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.536009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=250][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.548917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.559829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.573419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=128][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.584796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.590000] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:36.590928] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.590977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.591006] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.593649] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.594019] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=76] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.594075] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.594106] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:36.595773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.602948] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A65-0-0] [lt=78] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:36.611927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.612222] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83101][T1_L0_G0][T1][YB42AC10870A-000636671698E3CC-0-0] [lt=88][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525365794391, cache_obj->added_lc()=false, cache_obj->get_object_id()=53586, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:36.623320] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:36.623404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.634623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.645228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.657188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.657477] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=228] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:36.659699] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F08-0-0] [lt=24] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=34) [2025-05-31 02:58:36.660029] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678316658759000, start_id=1748678316658759000, end_id=1748678316658759001) [2025-05-31 02:58:36.662004] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=47] ====== traversal_flush timer task ====== [2025-05-31 02:58:36.662350] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B3-0-0] [lt=21] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678316543636000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:36.662515] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B3-0-0] [lt=156] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:36.667623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.678150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.690829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.701468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.712246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.716716] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=95] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:36.716775] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=50] temporary file module use no blocks [2025-05-31 02:58:36.716788] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:36.716804] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:36.716819] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525364898104, flush_io_finished_ret_:0, flush_io_finished_round_:19884, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:36.722696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.723188] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD36-0-0] [lt=34] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678316721700, keepalive_ts:-1}) [2025-05-31 02:58:36.723228] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD36-0-0] [lt=38] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678316721700, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:36.734758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.743535] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:36.743575] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=34] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:36.743589] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:36.743600] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:36.745019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.745112] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:36.747257] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [14078][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=59] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:36.746798] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:36.747712] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=909] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:36.750694] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=85, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:36.750779] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4569, avg_group_log_size=415) [2025-05-31 02:58:36.750802] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4102) [2025-05-31 02:58:36.750830] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:36.753775] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446D-0-0] [lt=79][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:36.755301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.755449] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4669, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:36.755473] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4569) [2025-05-31 02:58:36.755496] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=655, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:36.755545] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=111, avg_log_submit_to_flush_time=5663, avg_log_submit_to_slide_time=6074) [2025-05-31 02:58:36.755593] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:36.755780] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4569, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:36.765693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.776005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.786585] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.790262] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:36.798643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=239][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.808911] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.813714] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:36.813749] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=28] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:58:36.813774] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:36.821114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.824614] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=36][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:36.824663] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=44][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:36.824683] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 88FBE6E569360600, int: 1748678316850056}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 88FBE6E569360600, int: 1748678316850056}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:36.824848] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=67][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:36.825755] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:36.828898] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83078][T1_L0_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=31] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7385}, to_seq_no={branch:0, seq:4831}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382194} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:36.829001] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83078][T1_L0_G0][T1][YB42AC108714-000636671EAD63CD-0-0] [lt=59] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-05-31 02:58:36.831070] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:36.831204] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48662, [delay/us]:prepare:19, schedule:477, submit:474, rt:3370, total:4341) [2025-05-31 02:58:36.831226] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:36.831349] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=108] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48655, [delay/us]: prepare:19, schedule:477, submit:474, device:3370, total:4341) [2025-05-31 02:58:36.831365] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:36.831391] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4054, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:703, total:771) [2025-05-31 02:58:36.831455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:36.831493] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4054, [delay/us]: prepare:0, schedule:0, submit:600000000, device:703, total:771) [2025-05-31 02:58:36.831512] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:36.831527] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:36.831535] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:36.831544] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:36.831552] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:36.831561] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=290kB/s, net_out=160kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:36.831581] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399412][2nd=2810]}) [2025-05-31 02:58:36.831940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.845010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.847316] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CF-0-0] [lt=14][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:36.847373] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CF-0-0] [lt=52][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:36.847395] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CF-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f3, hex: 0B56E7E569360600, int: 1748678316873227}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fb, hex: 0B56E7E569360600, int: 1748678316873227}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:36.847585] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63CF-0-0] [lt=94][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:36.849475] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63CF-0-0] [lt=41] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7847}, to_seq_no={branch:0, seq:4885}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382195} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:36.856713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.862344] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC108714-000636671EAD63D0-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1748678316859033000, start_id=1748678316859033000, end_id=1748678316859033001) [2025-05-31 02:58:36.862891] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1748678316862627000, start_id=1748678316862627000, end_id=1748678316862627001) [2025-05-31 02:58:36.863476] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678316361981], gts:1748678316397868000, latest_srr:[mts=1748678316863424]}) [2025-05-31 02:58:36.866981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.869951] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678316869948, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006553, add_bytes: 0 [2025-05-31 03:58:36.871631] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [15659][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:36.871688] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:36.876515] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D1-0-0] [lt=65][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:36.876566] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D1-0-0] [lt=46][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:36.876587] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D1-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f0, hex: E8CBE7E569360600, int: 1748678316903400}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f8, hex: E8CBE7E569360600, int: 1748678316903400}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:36.876731] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D1-0-0] [lt=66][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:36.881619] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63D1-0-0] [lt=454] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8966}, to_seq_no={branch:0, seq:4970}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382196} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:36.881977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.891505] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC108714-000636671EAD63D2-0-0] [lt=16] get number(ret=0, service_type_=0, range=1, base_id=1748678316890470000, start_id=1748678316890470000, end_id=1748678316890470001) [2025-05-31 02:58:36.892304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.901493] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=31] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:36.902508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.904670] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83107][T1_L0_G0][T1][YB42AC108714-000636671EAD63D3-0-0] [lt=45][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:36.904718] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83107][T1_L0_G0][T1][YB42AC108714-000636671EAD63D3-0-0] [lt=42][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:36.904738] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83107][T1_L0_G0][T1][YB42AC108714-000636671EAD63D3-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099880c820, local_datums:0x2b099880c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b099880c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b099880c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4f0, hex: 9335E8E569360600, int: 1748678316930451}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4f8, hex: 9335E8E569360600, int: 1748678316930451}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:36.904952] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83107][T1_L0_G0][T1][YB42AC108714-000636671EAD63D3-0-0] [lt=67][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:36.907296] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC108714-000636671EAD63D3-0-0] [lt=36] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8535}, to_seq_no={branch:0, seq:6001}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382197} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:36.912743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.923348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.923498] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1748678316923495, bytes: 7127152885, bw: 0.366106 MB/s, add_ts: 1000428, add_bytes: 384054 [2025-05-31 02:58:36.929691] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63D4-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1748678316928745000, start_id=1748678316928745000, end_id=1748678316928745001) [2025-05-31 02:58:36.933648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.945714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.946976] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:36.950835] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D5-0-0] [lt=39][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:36.950935] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D5-0-0] [lt=95][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:36.950954] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D5-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: CDDBE8E569360600, int: 1748678316973005}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: CDDBE8E569360600, int: 1748678316973005}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:36.951112] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D5-0-0] [lt=90][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:36.953128] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC108714-000636671EAD63D5-0-0] [lt=48] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10728}, to_seq_no={branch:0, seq:6484}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382198} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:36.953542] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:36.956124] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=31] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:58:36.958961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.959640] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:36.969525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:36.969688] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [15659][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FE2-0-0] [lt=134] init point thread id with(&point=0x55c0908fbe80, idx_=3371, point=[thread id=15659, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=15659) [2025-05-31 02:58:36.981835] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A380D-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525366164002, cache_obj->added_lc()=false, cache_obj->get_object_id()=53587, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:36.993143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:36.999223] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83104][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33B3-0-0] [lt=183][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525366181394, cache_obj->added_lc()=false, cache_obj->get_object_id()=53588, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:36.999946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:37.003431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.004283] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=14][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.004317] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.004335] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: E369E9E569360600, int: 1748678317009379}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: E369E9E569360600, int: 1748678317009379}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 03085B0800000000, int: 140183555}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 00A0B20900000000, int: 162701312}]}) [2025-05-31 02:58:37.004565] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=152][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.008374] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=107] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:28086}, to_seq_no={branch:0, seq:17065}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382199} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.013994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.019026] INFO [STORAGE.TRANS] rollback_to_savepoint (ob_trans_part_ctx.cpp:8722) [83061][T1_L0_G0][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=57] rollback succeed trivially(trans_id={txid:11382199}, ls_id={id:1}, op_sn=3, to_scn={branch:0, seq:17065}, last_scn={branch:0, seq:17065}) [2025-05-31 02:58:37.021175] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81065][TimerWK3_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=40] runTimer to refresh locality_info(ret=0) [2025-05-31 02:58:37.021305] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2025-05-31 02:58:37.021456] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=143] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:37.024751] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:58:37.024976] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=70] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:58:37.024994] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=17] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:58:37.025017] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:58:37.025039] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=22] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:58:37.025055] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:58:37.025119] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=62] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:58:37.025132] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=8] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:58:37.025932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.026575] WDIAG [STORAGE.TRANS] handle_trans_msg_callback (ob_trans_service_v4.cpp:2487) [81426][pnio1][T1][YB42AC108714-000636671EAD63D7-0-0] [lt=26][errcode=0] handle trans msg callback(ret=0, elapsed_ts=0, tx_id={txid:11382199}, sender_ls_id={id:1}, receiver_ls_id={id:9223372036854775807}, msg_type=67, status=-4018, receiver_addr="172.16.135.20:2882", request_id=1748666364473311) [2025-05-31 02:58:37.026606] WDIAG [STORAGE.TRANS] process (ob_trans_rpc.h:288) [81426][pnio1][T0][YB42AC108714-000636671EAD63D7-0-0] [lt=27][errcode=0] trx rpc callback(ret=0, status=-4018, dst="172.16.135.20:2882", result={status:-4018, send_timestamp:1748678317019181, private_data:{val:18446744073709551615, v:3}}) [2025-05-31 02:58:37.028555] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA5C-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:37.030067] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44392448, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:37.031409] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=503312, clean_num=31457) [2025-05-31 02:58:37.036466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.040100] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2025-05-31 02:58:37.040264] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=29] gc stale ls task succ [2025-05-31 02:58:37.044542] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:37.051249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.054499] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=23][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae5d18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382200} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7570} write_epoch=1, prev=this=0x2b0980339510 trans_version={val:1748677891786141000, v:0} scn={val:1748677891786141000, v:0} tx_id={txid:11381612} prev=0x2b0988a86c48 next=(nil) modify_count=11 acc_checksum=228536282 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:33302} write_epoch=0, snapshot_version={val:1748678317051120000, v:0}, *this={this=0x2b09b31611a8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980339510 latest_compact_node=0x2b0980339510 max_trans_version={val:1748678256304880000, v:0} max_trans_id=11382116 max_elr_trans_version={val:1748678256304880000, v:0} max_elr_trans_id=11382116 latest_compact_ts=1748677891800321 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677891786141000, v:0} min_modify_scn={val:1748675827021622000, v:0}}) [2025-05-31 02:58:37.054556] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=56][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae5d18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382200} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7570} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678346949635, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678346949635, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317051120000, v:0}, tx_id:{txid:11382200}, scn:{branch:0, seq:7567}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382200}, tx_desc:{this:0x2b094e95a900, tx_id:{txid:11382200}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317054187, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317077890, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:7570}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.054696] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=138][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382200}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7570}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.054716] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382200} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.054746] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.054761] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 4C7DEAE569360600, int: 1748678317079884}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 4C7DEAE569360600, int: 1748678317079884}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 255E760000000000, int: 7757349}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0020B50000000000, int: 11870208}]}) [2025-05-31 02:58:37.054952] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=115][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.055635] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:37.057372] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83073][T1_L0_G0][T1][YB42AC108714-000636671EAD63D9-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9973}, to_seq_no={branch:0, seq:7568}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382200} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.063005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.067680] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83073][T1_L0_G0][T1][YB42AC108714-000636671EAD63DA-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1748678317065939000, start_id=1748678317065939000, end_id=1748678317065939001) [2025-05-31 02:58:37.073419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.083838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.087266] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DB-0-0] [lt=14][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.087314] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DB-0-0] [lt=43][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.087372] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DB-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f3, hex: 7AE4EAE569360600, int: 1748678317106298}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fb, hex: 7AE4EAE569360600, int: 1748678317106298}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.087543] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DB-0-0] [lt=131][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.089015] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83078][T1_L0_G0][T1][YB42AC108714-000636671EAD63DB-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14129}, to_seq_no={branch:0, seq:11645}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382201} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.094958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.106032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.111467] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83089][T1_L0_G0][T1][YB42AC108714-000636671EAD63DC-0-0] [lt=74] get number(ret=0, service_type_=0, range=1, base_id=1748678317111416000, start_id=1748678317111416000, end_id=1748678317111416001) [2025-05-31 02:58:37.116404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.124048] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=12][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.124086] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=33][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.124104] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f3, hex: 438CEBE569360600, int: 1748678317149251}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fb, hex: 438CEBE569360600, int: 1748678317149251}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.124270] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=87][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.127828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.132059] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=48] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11998}, to_seq_no={branch:0, seq:6622}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382202} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.139659] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=251][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.145054] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63DD-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700752544403076172e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633658293934377070e+03) [2025-05-31 02:58:37.149150] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FE6-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=16, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:37.149179] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FE6-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=5145, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:37.152153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.152619] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3ADB-0-0] [lt=69][errcode=-4006] not init(ret=-4006) [2025-05-31 02:58:37.152693] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3ADB-0-0] [lt=67] finish scan map holding items(scan_cnt=0) [2025-05-31 02:58:37.160723] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3ADC-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:37.161766] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678316948061000, v:0}, new_max_version={val:1748678316948061000, v:0}, last_min_version={val:1748678316658759000, v:0}, last_max_version={val:1748678316658759000, v:0}, total_time_us=9343, delta_us=213702) [2025-05-31 02:58:37.161832] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678316948061000, v:0}, min_version={val:1748678316948061000, v:0}, max_version={val:1748678316948061000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:37.162669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.173314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.173748] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [14998][T1_L0_G1][T1][YB42AC1087A7-00063665517F5422-0-0] [lt=79] init point thread id with(&point=0x55c0908f1940, idx_=2710, point=[thread id=14998, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=14998) [2025-05-31 02:58:37.174276] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=42][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae6658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382204} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:1933} write_epoch=1, prev=this=0x2b0980346330 trans_version={val:1748678256409415000, v:0} scn={val:1748678256409415000, v:0} tx_id={txid:11382120} prev=0x2b0988ad9850 next=(nil) modify_count=113 acc_checksum=1322425361 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:23026} write_epoch=0, snapshot_version={val:1748678317170032000, v:0}, *this={this=0x2b097cccada0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980346330 latest_compact_node=0x2b0980346330 max_trans_version={val:1748678256409415000, v:0} max_trans_id=11382120 max_elr_trans_version={val:1748678256409415000, v:0} max_elr_trans_id=11382120 latest_compact_ts=1748678256414153 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678256409415000, v:0} min_modify_scn={val:1748674855283998000, v:0}}) [2025-05-31 02:58:37.174405] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=127][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae6658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382204} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:1933} write_epoch=1, ctx={this:0x2b091cd511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347072644, tablet_id:{id:309}, table_iter:0x2b091cd4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347072644, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317170032000, v:0}, tx_id:{txid:11382204}, scn:{branch:0, seq:1930}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382204}, tx_desc:{this:0x2b0982b444e0, tx_id:{txid:11382204}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317173581, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317202297, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:1933}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.174570] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=162][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382204}, data:0x2b091cd4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:1933}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.174593] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382204} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.174624] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.174637] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b097ae0c820, local_datums:0x2b097ae0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b097ae0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b097ae0c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c4f1, hex: 7A5BECE569360600, int: 1748678317202298}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c4f9, hex: 7A5BECE569360600, int: 1748678317202298}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c501, hex: 543A710D00000000, int: 225524308}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b097ae0c509, hex: 543A710D00000000, int: 225524308}]}) [2025-05-31 02:58:37.174823] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=133][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.179184] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83082][T1_L0_G0][T1][YB42AC108714-000636671EAD63E1-0-0] [lt=50] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6965}, to_seq_no={branch:0, seq:1931}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382204} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.183651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.193859] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.201690] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:37.202062] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=57][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae6ab0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382205} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:1914} write_epoch=1, prev=this=0x2b0980346400 trans_version={val:1748678256468164000, v:0} scn={val:1748678256468164000, v:0} tx_id={txid:11382121} prev=0x2b0988ad9ca8 next=(nil) modify_count=113 acc_checksum=3165624472 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:14717} write_epoch=0, snapshot_version={val:1748678317199490000, v:0}, *this={this=0x2b09802c7738 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980346400 latest_compact_node=0x2b0980346400 max_trans_version={val:1748678256468164000, v:0} max_trans_id=11382121 max_elr_trans_version={val:1748678256468164000, v:0} max_elr_trans_id=11382121 latest_compact_ts=1748678256481655 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678256468164000, v:0} min_modify_scn={val:1748674855335943001, v:0}}) [2025-05-31 02:58:37.202146] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=81][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae6ab0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382205} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:1914} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347100303, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347100303, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317199490000, v:0}, tx_id:{txid:11382205}, scn:{branch:0, seq:1911}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382205}, tx_desc:{this:0x2b094e9f04c0, tx_id:{txid:11382205}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317200627, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317230556, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:1914}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.202404] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=254][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382205}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:1914}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.202455] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=48][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382205} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.202489] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=32][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.202574] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=77][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 09C8ECE569360600, int: 1748678317230089}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 09C8ECE569360600, int: 1748678317230089}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 5E08B90900000000, int: 163121246}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 5E08B90900000000, int: 163121246}]}) [2025-05-31 02:58:37.202806] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=175][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.204768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.205029] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63E3-0-0] [lt=37] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5962}, to_seq_no={branch:0, seq:1912}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382205} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.215196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.222003] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E5-0-0] [lt=124][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.222164] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E5-0-0] [lt=155][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.222216] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E5-0-0] [lt=44][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: D911EDE569360600, int: 1748678317248985}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: D911EDE569360600, int: 1748678317248985}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.222464] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E5-0-0] [lt=96][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.224204] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83106][T1_L0_G0][T1][YB42AC108714-000636671EAD63E5-0-0] [lt=112] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5985}, to_seq_no={branch:0, seq:3506}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382206} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.225431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.232555] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:37.235658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.244811] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E7-0-0] [lt=43][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.244937] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E7-0-0] [lt=121][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.244960] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E7-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 6471EDE569360600, int: 1748678317273444}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 6471EDE569360600, int: 1748678317273444}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.245219] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E7-0-0] [lt=95][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.245958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.246944] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC108714-000636671EAD63E7-0-0] [lt=152] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6189}, to_seq_no={branch:0, seq:2616}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382207} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.256882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.267208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.270083] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E9-0-0] [lt=43][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.270178] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E9-0-0] [lt=89][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.270201] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E9-0-0] [lt=15][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 52CEEDE569360600, int: 1748678317297234}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 52CEEDE569360600, int: 1748678317297234}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 00606F0A00000000, int: 175071232}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 00606F0A00000000, int: 175071232}]}) [2025-05-31 02:58:37.270466] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63E9-0-0] [lt=171][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.272606] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83104][T1_L0_G0][T1][YB42AC108714-000636671EAD63E9-0-0] [lt=80] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6547}, to_seq_no={branch:0, seq:3819}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382208} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.278137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.284371] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83094][T1_L0_G0][T1][YB42AC108714-000636671EAD63EA-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1748678317284239000, start_id=1748678317284239000, end_id=1748678317284239001) [2025-05-31 02:58:37.288436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.299501] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=54][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae75c8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382209} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4788} write_epoch=1, prev=this=0x2b0980339900 trans_version={val:1748677892106370000, v:0} scn={val:1748677892106370000, v:0} tx_id={txid:11381626} prev=0x2b0988a88450 next=(nil) modify_count=11 acc_checksum=2662266613 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:25566} write_epoch=0, snapshot_version={val:1748678317294208000, v:0}, *this={this=0x2b09b31623c0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980339900 latest_compact_node=0x2b0980339900 max_trans_version={val:1748678256604920000, v:0} max_trans_id=11382125 max_elr_trans_version={val:1748678256604920000, v:0} max_elr_trans_id=11382125 latest_compact_ts=1748677892115980 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677892106370000, v:0} min_modify_scn={val:1748675827281255000, v:0}}) [2025-05-31 02:58:37.299629] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=124][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae75c8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382209} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4788} write_epoch=1, ctx={this:0x2b091c8d31a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347195610, tablet_id:{id:309}, table_iter:0x2b091c8d16d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347195610, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317294208000, v:0}, tx_id:{txid:11382209}, scn:{branch:0, seq:4785}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382209}, tx_desc:{this:0x2b098d865ba0, tx_id:{txid:11382209}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317295375, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317322927, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4788}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.299789] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=158][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382209}, data:0x2b091c8d00c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4788}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.299812] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382209} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.300065] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=248][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.300086] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098040c820, local_datums:0x2b098040c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098040c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b098040c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c4f2, hex: B032EEE569360600, int: 1748678317322928}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c4fa, hex: B032EEE569360600, int: 1748678317322928}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c502, hex: 00B03F0100000000, int: 20951040}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098040c50a, hex: 00B03F0100000000, int: 20951040}]}) [2025-05-31 02:58:37.300224] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=81][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.301003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.303118] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83079][T1_L0_G0][T1][YB42AC108714-000636671EAD63EB-0-0] [lt=91] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10901}, to_seq_no={branch:0, seq:4786}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382209} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.311263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.315649] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83078][T1_L0_G0][T1][YB42AC108714-000636671EAD63EC-0-0] [lt=121] get number(ret=0, service_type_=0, range=1, base_id=1748678317314272000, start_id=1748678317314272000, end_id=1748678317314272001) [2025-05-31 02:58:37.321673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.333179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.334822] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=15117) [2025-05-31 02:58:37.341062] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63EF-0-0] [lt=16][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.341126] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63EF-0-0] [lt=59][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.341147] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63EF-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 09E3EEE569360600, int: 1748678317368073}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 09E3EEE569360600, int: 1748678317368073}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.341580] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63EF-0-0] [lt=67][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.343569] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83099][T1_L0_G0][T1][YB42AC108714-000636671EAD63EF-0-0] [lt=6] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5726}, to_seq_no={branch:0, seq:3423}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382211} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.344085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.353236] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=30] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:120K/26:341939 disk:120K/26:341939 t1m2:120K/26:341939 t1m2g0:120K/26:341939 |q0: 0 |q1: 0 [2025-05-31 02:58:37.354549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.359254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.359297] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.359325] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.359939] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.363830] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=15][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae7b28 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382212} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3074} write_epoch=1, prev=this=0x2b09802c7c30 trans_version={val:1748674855578654000, v:0} scn={val:1748674855578654000, v:0} tx_id={txid:11377462} prev=0x2b09b3162c20 next=(nil) modify_count=1 acc_checksum=1506457104 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:25777} write_epoch=3, snapshot_version={val:1748678317361268000, v:0}, *this={this=0x2b09b3162ac0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b09802c7c30 latest_compact_node=(nil) max_trans_version={val:1748678256679690000, v:0} max_trans_id=11382128 max_elr_trans_version={val:1748678256679690000, v:0} max_elr_trans_id=11382128 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748674855578654000, v:0} min_modify_scn={val:1748674855578654000, v:0}}) [2025-05-31 02:58:37.363955] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=122][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae7b28 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382212} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3074} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347262264, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347262264, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317361268000, v:0}, tx_id:{txid:11382212}, scn:{branch:0, seq:3071}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382212}, tx_desc:{this:0x2b094e9cde60, tx_id:{txid:11382212}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317362547, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317391359, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:3074}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.364061] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=105][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382212}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3074}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.364080] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382212} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.364110] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.364123] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: C143EFE569360600, int: 1748678317392833}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: C143EFE569360600, int: 1748678317392833}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0060090000000000, int: 614400}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0060090000000000, int: 614400}]}) [2025-05-31 02:58:37.364342] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=118][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.364669] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:37.365676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.366998] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63F1-0-0] [lt=14] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7084}, to_seq_no={branch:0, seq:3072}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382212} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.376382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.387106] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=74][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae7da8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382213} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5658} write_epoch=1, prev=this=0x2b0980346598 trans_version={val:1748678256714829000, v:0} scn={val:1748678256714829000, v:0} tx_id={txid:11382129} prev=0x2b097cd42c98 next=(nil) modify_count=113 acc_checksum=2632428522 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:19577} write_epoch=0, snapshot_version={val:1748678317383943000, v:0}, *this={this=0x2b09b3162d68 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980346598 latest_compact_node=0x2b0980346598 max_trans_version={val:1748678256714829000, v:0} max_trans_id=11382129 max_elr_trans_version={val:1748678256714829000, v:0} max_elr_trans_id=11382129 latest_compact_ts=1748678256725728 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678256714829000, v:0} min_modify_scn={val:1748674855636178000, v:0}}) [2025-05-31 02:58:37.387186] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=77][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae7da8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382213} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5658} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347282367, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347282367, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317383943000, v:0}, tx_id:{txid:11382213}, scn:{branch:0, seq:5655}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382213}, tx_desc:{this:0x2b094e875ea0, tx_id:{txid:11382213}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317385350, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317411112, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:5658}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.387283] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=95][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382213}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:5658}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.387301] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382213} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.387331] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.387346] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 6390EFE569360600, int: 1748678317412451}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 6390EFE569360600, int: 1748678317412451}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0010380000000000, int: 3674112}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0010380000000000, int: 3674112}]}) [2025-05-31 02:58:37.387575] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=173][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.388105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.389057] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC108714-000636671EAD63F3-0-0] [lt=75] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9055}, to_seq_no={branch:0, seq:5656}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382213} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.398561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.398711] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83074][T1_L0_G0][T1][YB42AC108714-000636671EAD63F4-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678317397311000, start_id=1748678317397311000, end_id=1748678317397311001) [2025-05-31 02:58:37.401983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:37.409045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.419261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.426236] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F7-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.426293] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F7-0-0] [lt=51][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.426313] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F7-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 882BF0E569360600, int: 1748678317452168}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 882BF0E569360600, int: 1748678317452168}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 00C0BC0000000000, int: 12369920}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 00C0BC0000000000, int: 12369920}]}) [2025-05-31 02:58:37.426480] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F7-0-0] [lt=88][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.429492] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.429951] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63F7-0-0] [lt=20] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6998}, to_seq_no={branch:0, seq:4717}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382215} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.434325] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:37.440720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.440908] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83100][T1_L0_G0][T1][YB42AC108714-000636671EAD63F8-0-0] [lt=97] get number(ret=0, service_type_=0, range=1, base_id=1748678317440069000, start_id=1748678317440069000, end_id=1748678317440069001) [2025-05-31 02:58:37.447613] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:37.447656] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae8648 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382216} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:2861} write_epoch=1, prev=this=0x2b0988a89138 trans_version={val:1748677892366043000, v:0} scn={val:1748677892366043000, v:0} tx_id={txid:11381633} prev=0x2b0988a88f80 next=(nil) modify_count=9 acc_checksum=2035094073 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678316749386000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:43911} write_epoch=3, snapshot_version={val:1748678317441608000, v:0}, *this={this=0x2b09802c7d90 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b0988a89138 latest_compact_node=0x2b0980319770 max_trans_version={val:1748678256781422000, v:0} max_trans_id=11382132 max_elr_trans_version={val:1748678256781422000, v:0} max_elr_trans_id=11382132 latest_compact_ts=1748677042082843 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677892366043000, v:0} min_modify_scn={val:1748675827437572000, v:0}}) [2025-05-31 02:58:37.447716] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=57][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae8648 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382216} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:2861} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678347345940, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678347345940, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678317441608000, v:0}, tx_id:{txid:11382216}, scn:{branch:0, seq:2858}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382216}, tx_desc:{this:0x2b094e808a60, tx_id:{txid:11382216}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678317446431, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678317475559, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:2861}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:37.447952] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=233][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382216}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:2861}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:37.447975] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382216} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:37.448006] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.448020] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 478BF0E569360600, int: 1748678317476679}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 478BF0E569360600, int: 1748678317476679}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 00908A0000000000, int: 9080832}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 00908A0000000000, int: 9080832}]}) [2025-05-31 02:58:37.448161] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=85][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.449369] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC108714-000636671EAD63F9-0-0] [lt=62] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5174}, to_seq_no={branch:0, seq:2859}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382216} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.452138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.453719] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:37.456714] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71] get_max_applied_scn(scn={val:1748678317353970000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137145617}, palf_committed_end_scn_:{val:1748678317440069000, v:0}, last_check_scn_:{val:1748678317440069000, v:0}, max_applied_cb_scn_:{val:1748678317353970000, v:0}}) [2025-05-31 02:58:37.456770] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137145617}, readable_log_tail:{lsn:2137145617}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10206959, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:37.462280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.472983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.474357] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63FB-0-0] [lt=38][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:37.474404] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63FB-0-0] [lt=43][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:37.474426] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63FB-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c620c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f0, hex: B4E4F0E569360600, int: 1748678317499572}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f8, hex: B4E4F0E569360600, int: 1748678317499572}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:37.474701] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671EAD63FB-0-0] [lt=173][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:37.476008] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671EAD63FB-0-0] [lt=406] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7988}, to_seq_no={branch:0, seq:5543}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382217} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:37.483328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.493605] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.505389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.515616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.525904] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.536305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.546601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.557048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.569730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.574057] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81060][TimerWK0_ServerTracerTim][T0][YB42AC1087A6-0006366547BA5934-0-0] [lt=60] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:37.579939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.588649] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:37.588693] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:37.588732] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:37.588758] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:37.590258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.591047] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.591091] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.591178] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.591362] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.591529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.591566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.594527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:37.600309] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9] start to print blacklist info [2025-05-31 02:58:37.600902] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=29] ls blacklist refresh finish(cost_time=33060) [2025-05-31 02:58:37.601137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.602205] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:37.611403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.616319] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=55] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-05-31 02:58:37.621536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.633553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.635029] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:37.643798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.654036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.664299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.674938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.685141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.695357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.707381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.718010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.730288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.740474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.750785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.756767] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446E-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:37.759677] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=156] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:37.759731] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=31, avg_log_batch_cnt=1, total_group_log_size=18693, avg_group_log_size=603) [2025-05-31 02:58:37.759753] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=18693) [2025-05-31 02:58:37.759768] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:37.760904] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=2008, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:37.760931] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=18693) [2025-05-31 02:58:37.760953] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=494, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:37.761041] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT FLUSH CB COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=26, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:37.761137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=158][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.761924] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FFA-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=31, avg_log_gen_to_freeze_time=28, avg_log_gen_to_submit_time=142, avg_log_submit_to_flush_time=2549, avg_log_submit_to_slide_time=4151) [2025-05-31 02:58:37.762089] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=18693, hit_cnt=32, read_cnt=32, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:37.771445] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.774992] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=60] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:37.784295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.794574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.804274] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:37.804969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.815268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.825563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.836449] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:37.836570] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:37.836618] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:30, norm_iops:8, bw:138503, [delay/us]:prepare:18, schedule:229, submit:53, rt:1653, total:1955) [2025-05-31 02:58:37.836636] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:37.836712] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:30, avg_bw:138521, [delay/us]: prepare:18, schedule:229, submit:53, device:1653, total:1955) [2025-05-31 02:58:37.836727] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:37.836753] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:771, total:787) [2025-05-31 02:58:37.836779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:37.836809] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:771, total:787) [2025-05-31 02:58:37.836827] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:37.836841] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:37.836850] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:37.836859] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:37.836924] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:37.836939] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=772kB/s, net_out=367kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:37.836959] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399413][2nd=2669]}) [2025-05-31 02:58:37.837120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.847631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.859016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.869260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.869384] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:37.870992] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678317870987, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001039, add_bytes: 0 [2025-05-31 02:58:37.871045] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [14998][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=53] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:37.879639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.890095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.900533] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3806-0-0] [lt=11] [APPLY STAT CB APPEND COST TIME](cur_stat_count=64, stat_interval=5000000, avg (count/cost)=169, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:37.900571] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3806-0-0] [lt=38] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=64, stat_interval=5000000, avg (count/cost)=6429, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:37.900580] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3806-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=64, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:37.900587] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3806-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=64, stat_interval=5000000, avg (count/cost)=-1021, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:37.900594] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3806-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=64, stat_interval=5000000, avg (count/cost)=5577, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:37.900675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.903132] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=44] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:37.911135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=156][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.921617] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=51] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:37.923599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.932170] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1748678317932163, bytes: 7127869086, bw: 0.677153 MB/s, add_ts: 1008668, add_bytes: 716201 [2025-05-31 02:58:37.933808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.944085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.947989] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:37.956716] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:37.956812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.967042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.977317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.987609] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:37.989775] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A380F-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525367171943, cache_obj->added_lc()=false, cache_obj->get_object_id()=53589, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:37.998236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.005595] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:38.008494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.019200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.029547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.031366] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2060-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:38.031549] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2060-0-0] [lt=168] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=186) [2025-05-31 02:58:38.037286] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:38.039819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.041188] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2025-05-31 02:58:38.041258] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=36] gc stale ls task succ [2025-05-31 02:58:38.044821] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:38.048770] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA5D-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:38.054005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.058456] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:38.064231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.074731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.084941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.096067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.106231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.117518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.127984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=128][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.138171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.148479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.158770] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.163504] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF90F-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.724751949310302734e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.590146127404331310e+03) [2025-05-31 02:58:38.167763] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FFE-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=29, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:38.167789] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC0FFE-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=4737, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:38.168493] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678317959073000, v:0}, new_max_version={val:1748678317959073000, v:0}, last_min_version={val:1748678317557267001, v:0}, last_max_version={val:1748678317557267001, v:0}, total_time_us=7064, delta_us=209417) [2025-05-31 02:58:38.168533] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678317959073000, v:0}, min_version={val:1748678317959073000, v:0}, max_version={val:1748678317959073000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:38.168555] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:38.169016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.173943] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=33] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:38.179302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.191451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.201664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.209105] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:38.212495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.222731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.232951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.240633] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:38.244831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.255099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.265914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.270075] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4161-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678318266773000, start_id=1748678318266773000, end_id=1748678318266773001) [2025-05-31 02:58:38.276207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.278487] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=128] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:38.278514] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:38.278560] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1099-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:38.278589] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA1099-0-0] [lt=25] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:38.286475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.291037] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A563F-0-0] [lt=41] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678313557774, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:38.291089] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A563F-0-0] [lt=50] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678313560199, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:38.291115] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A563F-0-0] [lt=20] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678313565274, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678313565279, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:38.297404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.300647] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81744][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=50] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:38.308003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.318219] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.328630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.340695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.342804] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=7982) [2025-05-31 02:58:38.350943] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.352382] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3811-0-0] [lt=92][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525367534550, cache_obj->added_lc()=false, cache_obj->get_object_id()=53590, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:38.355032] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3813-0-0] [lt=47][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:38.358262] WDIAG get_refactored (ob_hashtable.h:1120) [83091][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3815-0-0] [lt=31][errcode=-4201] hashtable not init [2025-05-31 02:58:38.358939] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.359362] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3816-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678318358544000, start_id=1748678318358544000, end_id=1748678318358544001) [2025-05-31 02:58:38.359375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.359492] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.359522] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.359873] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:72K/16:73047 disk:72K/16:73047 t1m2:72K/16:73047 t1m2g0:72K/16:73047 |q0: 0 |q1: 0 [2025-05-31 02:58:38.363798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.364985] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=11] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:38.368971] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=59] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678318368937], gts:1748678318368242000, latest_srr:[mts=1748678318368937]}) [2025-05-31 02:58:38.369009] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:38.374115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.384379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.396612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.408086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.409328] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:38.418367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.428589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.441150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.441972] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:38.451121] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:38.451422] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.459112] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:38.461640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.471892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.482078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.492314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.503300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.506256] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C5-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:38.514147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.524360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.526016] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC10871E-000636654E089A21-0-0] [lt=68][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525367708186, cache_obj->added_lc()=false, cache_obj->get_object_id()=53591, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:38.534584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.545083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.555315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.565578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.578599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.588986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=213][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.589931] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.590330] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=122] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.590614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.590663] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.593604] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=65] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.593717] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.593750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:38.600428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.610597] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=157280, replace_num=15728) [2025-05-31 02:58:38.610761] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=160] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:38.611121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.623627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.633953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.642533] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:38.646701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.657002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.667365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.668510] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=43] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:38.668567] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=41] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=984) [2025-05-31 02:58:38.678553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.689301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.699526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.709740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.717512] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=85] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:38.717600] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=79] temporary file module use no blocks [2025-05-31 02:58:38.717611] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:38.717626] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:38.717641] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525366899312, flush_io_finished_ret_:0, flush_io_finished_round_:19886, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:38.720066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=136][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.731140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.741624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.752440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.759525] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A446F-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:38.764123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.768157] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=75, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:38.768200] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4547, avg_group_log_size=413) [2025-05-31 02:58:38.768221] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4547) [2025-05-31 02:58:38.768236] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:38.777191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.778454] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3451, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:38.778487] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4547) [2025-05-31 02:58:38.778511] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=271, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:38.778577] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=44, avg_log_gen_to_submit_time=139, avg_log_submit_to_flush_time=3797, avg_log_submit_to_slide_time=6352) [2025-05-31 02:58:38.778610] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=45, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:38.778752] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4547, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:38.787698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.801158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.810975] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:38.812214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.821987] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:38.822096] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=42] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:58:38.822126] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=20] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:38.823655] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:38.823848] WDIAG read_multi_row (ob_table_access_helper.h:238) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=56][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:38.826403] INFO delete_row (ob_table_access_helper.h:339) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=199] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678318824049) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678318824049) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:58:38.826514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.836733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.843121] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=129] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:38.843471] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:38.843618] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48812, [delay/us]:prepare:16, schedule:536, submit:117, rt:2608, total:3278) [2025-05-31 02:58:38.843638] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:38.843696] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48811, [delay/us]: prepare:16, schedule:536, submit:117, device:2608, total:3278) [2025-05-31 02:58:38.843709] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:38.843780] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4067, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:509, total:522) [2025-05-31 02:58:38.843807] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:38.843893] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=78] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4067, [delay/us]: prepare:0, schedule:0, submit:600000000, device:509, total:522) [2025-05-31 02:58:38.843914] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:38.843930] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:38.843948] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:38.844038] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=85] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:38.844048] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:38.844066] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SERVER] net_in=160kB/s, net_out=102kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:38.844105] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] successfully alloc block(macro_id={[ver=1,mode=0,seq=399414][2nd=2951]}) [2025-05-31 02:58:38.846920] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.857208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.867489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.877789] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678318877784, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006797, add_bytes: 0 [2025-05-31 02:58:38.877908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.877944] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:38.877991] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [11960][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:58:38.882506] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81742][T1_TimerWK1_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=51] clear dtl interm result cost(us)(clear_cost=4133, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:58:38.895105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.903806] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=40] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:38.905629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.916323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.928079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.928202] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:58:38.929057] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=26] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:58:38.936509] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1748678318936504, bytes: 7128033040, bw: 0.155683 MB/s, add_ts: 1004341, add_bytes: 163954 [2025-05-31 02:58:38.938662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.949074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.951421] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:38.959299] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:38.959399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.969656] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.980071] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B99-0-0] [lt=9] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678319011005, keepalive_ts:-1}) [2025-05-31 02:58:38.980118] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B99-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678319011005, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:38.980439] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:38.991217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.000799] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3817-0-0] [lt=60][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525368182966, cache_obj->added_lc()=false, cache_obj->get_object_id()=53592, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:39.001669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.011939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.012289] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=65] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:39.021680] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=47] ====== check clog disk timer task ====== [2025-05-31 02:58:39.021735] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=47] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:39.022521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.032785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.041395] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-05-31 02:58:39.041528] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=15] gc stale ls task succ [2025-05-31 02:58:39.043693] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:39.044822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.045186] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=534769, clean_num=31457) [2025-05-31 02:58:39.048051] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=212] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:39.060060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.060443] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:39.068198] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA5E-0-0] [lt=34][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:39.073136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.083515] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.096728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=142][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.107124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.117971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.128358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.138781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.149665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=158][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.162761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.168915] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF91C-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700752544403076172e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633658293934377070e+03) [2025-05-31 02:58:39.173089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.174109] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=11, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:39.174137] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5459, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:39.174546] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678318968288000, v:0}, new_max_version={val:1748678318968288000, v:0}, last_min_version={val:1748678318567276001, v:0}, last_max_version={val:1748678318567276001, v:0}, total_time_us=7981, delta_us=206254) [2025-05-31 02:58:39.174581] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678318968288000, v:0}, min_version={val:1748678318968288000, v:0}, max_version={val:1748678318968288000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:39.184353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.194615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.205248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.212635] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:39.216993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.227328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.238197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.238371] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0D16-0-0] [lt=55] handle gts request statistics(total_rt=1597, total_cnt=195, avg_rt=8.147959183673469497e+00, avg_cnt=1.950000000000000000e+01) [2025-05-31 02:58:39.245712] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:39.250359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.260565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.265675] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=33] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:39.266827] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678319168360000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678319266776}, version_delta=98464) [2025-05-31 02:58:39.271489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.284089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.284690] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=78] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:39.286649] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=27] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:39.286772] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=32] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678312108758000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:39.294378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.304709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.314971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.326634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.336905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.348161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.351101] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3819-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525368533269, cache_obj->added_lc()=false, cache_obj->get_object_id()=53593, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:39.354687] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC100A-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4121) [2025-05-31 02:58:39.358717] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.359188] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.359228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.359256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.359282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.368945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.370828] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=63] ts response task statistics(alloc_count=485810, free_count=242904) [2025-05-31 02:58:39.371013] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:39.371096] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:39.371115] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678318870031], gts:1748678318902468000, latest_srr:[mts=1748678319371035]}) [2025-05-31 02:58:39.377636] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=142] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:39.379206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.382467] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=97] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-05-31 02:58:39.389468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.399821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.410549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=214][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.412886] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:39.421000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.432551] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.442781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.446217] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:39.452334] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:39.454088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.459565] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:39.464218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.474658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.485480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.497691] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672F3E3F87-0-0] [lt=26] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=283, Holding Memory(MB)=134, Tenant Remain Memory(MB)=249, Usable Remain Memory(MB)=149, Last Update Limit Timestamp=1748678319495185, Is Updated=true) [2025-05-31 02:58:39.498460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.498771] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=36] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:614005 disk:48K/11:614005 t1m2:48K/11:614005 t1m2g0:48K/11:614005 |q0: 0 |q1: 0 [2025-05-31 02:58:39.508591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.519982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.530406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.540649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.553562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.564105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.567483] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] no need to update min/max version(new_version={val:1748678318968288001, v:0}, min_version={val:1748678318968288001, v:0}, max_version={val:1748678318968288001, v:0}) [2025-05-31 02:58:39.574363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.585407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.590953] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591710] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=88] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591943] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.591987] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:39.596306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.597325] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A66-0-0] [lt=36] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:39.606578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.613285] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=85] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:39.616783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.627277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.639375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.649148] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:39.649592] INFO [COMMON] wash (ob_kvcache_store.cpp:355) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash time detail, (compute_wash_size_time=181, refresh_score_time=106, wash_time=0) [2025-05-31 02:58:39.650046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.653236] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC108714-000636671E4D5DE0-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678319650169000, start_id=1748678319650169000, end_id=1748678319650169001) [2025-05-31 02:58:39.660687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.667643] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678319367241000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=300385, in_cluster_service=true, cluster_version={val:1748678318968288001, v:0}, min_cluster_version={val:1748678318968288001, v:0}, max_cluster_version={val:1748678318968288001, v:0}, get_cluster_version_err=0, cluster_version_delta=699338, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195043, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=310612868, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:39.672016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.685831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.696194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.704304] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678309698383, end_time=1748678319704301, dag_status={}) [2025-05-31 02:58:39.704405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=51] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704445] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=39] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704456] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704463] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704498] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:39.704509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:58:39.704518] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:58:39.708470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.718665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.728977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.739292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.749837] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.762820] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4470-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:39.763688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.774085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.784481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.794805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.805206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=204][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.813972] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:39.815450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.819793] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] BlockGCTimerTask success(ret=0, cost_time_us=20, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:58:39.826145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.836550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.846779] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.850151] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:39.856836] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:39.856968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=93] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44460, [delay/us]:prepare:13, schedule:509, submit:150, rt:2409, total:3082) [2025-05-31 02:58:39.856987] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:39.857088] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=85] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44460, [delay/us]: prepare:13, schedule:509, submit:150, device:2409, total:3082) [2025-05-31 02:58:39.857103] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:39.857129] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4042, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:343, total:355) [2025-05-31 02:58:39.857155] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:39.857184] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4042, [delay/us]: prepare:0, schedule:0, submit:600000000, device:343, total:355) [2025-05-31 02:58:39.857202] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:39.857216] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:39.857224] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:39.857232] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:39.857241] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:39.857249] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=152kB/s, net_out=94kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:39.857270] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399415][2nd=2528]}) [2025-05-31 02:58:39.857484] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:39.857534] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4487, avg_group_log_size=407) [2025-05-31 02:58:39.857555] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4954) [2025-05-31 02:58:39.857568] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:39.858095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.864373] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=47][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:39.865421] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4114, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:39.865442] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4487) [2025-05-31 02:58:39.865465] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=372, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:39.865554] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=100, avg_log_submit_to_flush_time=4925, avg_log_submit_to_slide_time=5478) [2025-05-31 02:58:39.865591] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=49, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:39.865631] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4487, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:39.868399] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=42] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19340374016, fast_delete_log_mode=false) [2025-05-31 02:58:39.871173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.874248] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678319371387], gts:1748678319404356000, latest_srr:[mts=1748678319874217]}) [2025-05-31 02:58:39.879486] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=96] PNIO [ratelimit] time: 1748678319879482, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001698, add_bytes: 0 [2025-05-31 02:58:39.884159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.884238] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:39.884984] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [15355][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=23] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-05-31 02:58:39.894567] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:74, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183562814810, recv_hp_rpc_cnt:74832, recv_np_rpc_cnt:876574, recv_lp_rpc_cnt:0, recv_mysql_cnt:46939, recv_task_cnt:2470, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1230123 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9865, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:543296, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1748678319884233}{group_id:10, queue_size:0, recv_req_cnt:103793, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18333, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678313738587}{group_id:2, queue_size:0, recv_req_cnt:150707, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:100949, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:107756, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272102, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=693 pcode=0x717:cnt=491 pcode=0x1502:cnt=260 pcode=0x710:cnt=175 pcode=0x523:cnt=100, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:58:39.895730] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1161] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:83229861, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:58:39.896475] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=742] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84749022, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:58:39.897372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=885][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.906012] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=28] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:39.907612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.917942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=151][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.930462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.936614] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1748678319936610, bytes: 7128197717, bw: 0.157032 MB/s, add_ts: 1000106, add_bytes: 164677 [2025-05-31 02:58:39.940776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.951771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.953066] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:39.962228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.962636] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=134] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:39.964537] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=107] ObArchiveSender is running(thread_index=0) [2025-05-31 02:58:39.972951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.978407] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB9462D-0-0] [lt=11][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:2490379960, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678320004024, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1379781, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678319975694, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678319978402, enqueue_timestamp_:0, request_arrival_time_:1748678319978402, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:39.978516] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB9462D-0-0] [lt=87][errcode=-5150] deliver rpc request fail(&req=0x2b09a9204098, ret=-5150) [2025-05-31 02:58:39.983811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:39.994685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.005045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.009725] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A381B-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525369191893, cache_obj->added_lc()=false, cache_obj->get_object_id()=53594, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.014134] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:40.015170] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137157657}], scn_array=[cnt:1, {val:1748678319990943000, v:0}], curr_size=548, accum_size=1137, time_cost=21247) [2025-05-31 02:58:40.015338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.026172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.037551] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.042983] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=67] current monitor number(seq_=-1) [2025-05-31 02:58:40.043146] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=71] gc stale ls task succ [2025-05-31 02:58:40.048015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.051450] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:40.055249] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=51] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:40.055792] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2061-0-0] [lt=62] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:40.055827] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2061-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=36) [2025-05-31 02:58:40.060916] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.062788] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=46] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:40.071236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.082282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.087544] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81742][T1_TimerWK1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=156] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:40.093083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.095438] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83087][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA5F-0-0] [lt=64][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:40.105359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.116116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.118070] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:165) [83004][T1_ArcSeq][T1][YB42AC1087A6-000636654269FC95-0-0] [lt=33] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-05-31 02:58:40.128355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.141630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.151923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.162433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.174581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.179701] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3ADE-0-0] [lt=19] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:40.184037] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA67-0-0] [lt=58][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525369366206, cache_obj->added_lc()=false, cache_obj->get_object_id()=53595, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.184705] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AE0-0-0] [lt=33] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:40.185522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.185655] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=35] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:40.195749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.206555] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.215644] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:40.217291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.229427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.239674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.252271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.255652] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:40.265173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:40.267147] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [16802][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:40.267201] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-05-31 02:58:40.267851] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:40.268196] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=343] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=8058, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:40.272407] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF92B-0-0] [lt=74] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.724751949310302734e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.590146127404331310e+03) [2025-05-31 02:58:40.272727] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] get number(ret=0, service_type_=0, range=1, base_id=1748678320267743000, start_id=1748678320267743000, end_id=1748678320267743001) [2025-05-31 02:58:40.278113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.280332] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678319990943000, v:0}, new_max_version={val:1748678319990943000, v:0}, last_min_version={val:1748678319868819000, v:0}, last_max_version={val:1748678319868819000, v:0}, total_time_us=9873, delta_us=289386) [2025-05-31 02:58:40.280385] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678319990943000, v:0}, min_version={val:1748678319990943000, v:0}, max_version={val:1748678319990943000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:40.280578] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=32] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:40.280604] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:40.280696] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3A-0-0] [lt=16] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:40.280750] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3A-0-0] [lt=50] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:40.290000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.301507] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.312250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.322991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.333405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.345177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.356920] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.360964] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.361061] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.361099] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.361178] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.363783] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81064][TimerWK2_AutoLSLocRpc][T0][YB42AC1087A6-0006366542DA5457-0-0] [lt=11] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:58:40.367965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.374802] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=738, get_gts_with_stc_cnt=56, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=56, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:40.374926] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=119] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:40.378292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.387526] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] Garbage Collector is running(seq_=1992, gc_interval=10000000) [2025-05-31 02:58:40.387682] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=125] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:58:40.387708] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=16] gc_check_member_list_ cost time(ret=0, time_us=56) [2025-05-31 02:58:40.387721] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:58:40.387734] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:58:40.387778] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=39] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:58:40.387802] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=21] execute_gc cost time(ret=0, time_us=26) [2025-05-31 02:58:40.387811] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-05-31 02:58:40.387819] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:58:40.387838] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:58:40.388538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.398953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.403068] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=29] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:40.409249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.417826] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:40.421992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.428589] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D511F-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525369610758, cache_obj->added_lc()=false, cache_obj->get_object_id()=53596, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.434235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.444428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.453313] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-31 02:58:40.454647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.457371] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=134] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:40.462804] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:40.463477] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:738876 disk:48K/11:738876 t1m2:48K/11:738876 t1m2g0:48K/11:738876 |q0: 0 |q1: 0 [2025-05-31 02:58:40.466226] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [14466][T1_L0_G1][T1][YB42AC10871E-0006366549CC1016-0-0] [lt=44] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5502) [2025-05-31 02:58:40.467814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.477502] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:40.477590] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] get wrs ts(ls_id={id:1}, delta=208832, log_service_ts=1748678320267743, min_tx_service_ts=4611686018427387, timestamp={val:1748678320267743000, v:0}) [2025-05-31 02:58:40.477614] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=60, Other_Error_Cnt=0, Submit_Succ_Cnt=39, last_scn="{val:1748678315524251001, v:0}", last_lsn={lsn:2137124593}, last_gts={val:1748678320373543000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:58:40.478277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.488561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.498887] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.509372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.514361] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C6-0-0] [lt=37] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:40.518739] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=58] ====== checkpoint timer task ====== [2025-05-31 02:58:40.519596] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:40.519642] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=46] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:40.520128] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=88] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:40.520218] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:40.520338] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=112] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:40.520355] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1748678313702670001, v:0}) [2025-05-31 02:58:40.520363] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:40.520370] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:40.520384] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:40.520405] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:40.520419] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:40.520440] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:40.520462] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:40.520472] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81742][T1_TimerWK1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:40.520745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.531139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.542288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.552574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.565558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.580447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.590934] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=103] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.591042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.591701] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.591742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.592506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.592662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.592692] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.592779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:40.597509] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:40.597547] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.597583] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.597605] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:40.602739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.610421] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] ls blacklist refresh finish(cost_time=42138) [2025-05-31 02:58:40.615057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.618322] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=348] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:40.625319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.635615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.647008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.657329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.658019] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=138] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:40.666008] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:40.666055] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] try gc retain ctx [2025-05-31 02:58:40.666063] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-31 02:58:40.666168] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678320473655000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678310455421001, v:0}, keep_alive_scn_:{val:1748678315524251001, v:0}, update_ts_:1748678320664814}) [2025-05-31 02:58:40.666779] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:40.667829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.678317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.690930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.702503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.712903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.718410] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:40.718503] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=83] temporary file module use no blocks [2025-05-31 02:58:40.718518] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:40.718535] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:40.718589] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=52] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525368900202, flush_io_finished_ret_:0, flush_io_finished_round_:19888, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:40.723224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.736368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.739658] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC10871E-000636654DFA0D1F-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1748678320739185000, start_id=1748678320739185000, end_id=1748678320739185001) [2025-05-31 02:58:40.745917] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032025, max_lsn:2137160918, max_scn:{val:1748678320675578000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137160918}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095217878}}, last_submit_lsn:{lsn:2137160329}, last_submit_end_lsn:{lsn:2137160918}, last_submit_log_id:4032025, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137160329}, max_flushed_end_lsn:{lsn:2137160918}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137160918}, last_slide_log_id:4032025, last_slide_scn:{val:1748678320675578000, v:0}, last_slide_lsn:{lsn:2137160329}, last_slide_end_lsn:{lsn:2137160918}, last_slide_log_pid:11, last_slide_log_accum_checksum:187985148, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032026, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137160918}, readable_log_tail:{lsn:2137160918}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10191658, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:58:40.746245] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=318] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678320682248, last_flushed_end_lsn:{lsn:2137160918}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678320678279, last_flushed_end_lsn:{lsn:2137160918}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678320683276, last_flushed_end_lsn:{lsn:2137160918}}]) [2025-05-31 02:58:40.746272] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032026, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678320682248, last_flushed_end_lsn:{lsn:2137160918}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678320678279, last_flushed_end_lsn:{lsn:2137160918}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678320683276, last_flushed_end_lsn:{lsn:2137160918}}]) [2025-05-31 02:58:40.746696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.756102] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [8165][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD37-0-0] [lt=19] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678320752676, keepalive_ts:-1}) [2025-05-31 02:58:40.756142] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [8165][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD37-0-0] [lt=37] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678320752676, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:40.757001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.766250] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4471-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:40.767928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.778513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.789418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.800101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.807751] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=6] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137160918}, readable_log_tail:{lsn:2137160918}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10191658, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:40.808031] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=277] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:58:40.808067] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=16] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:58:40.810341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.819432] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=173008, replace_num=15728) [2025-05-31 02:58:40.819471] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:40.821265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.833751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.845961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.846633] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:40.846719] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=77] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:58:40.846772] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=42] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:40.856534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.860222] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:40.866886] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:40.866951] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44610, [delay/us]:prepare:40, schedule:539, submit:513, rt:4536, total:5630) [2025-05-31 02:58:40.866970] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:40.867032] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44612, [delay/us]: prepare:40, schedule:539, submit:513, device:4536, total:5630) [2025-05-31 02:58:40.867045] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:40.867170] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4055, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:553, total:568) [2025-05-31 02:58:40.867200] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:40.867259] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4055, [delay/us]: prepare:0, schedule:0, submit:600000000, device:553, total:568) [2025-05-31 02:58:40.867278] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:40.867294] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:40.867303] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:40.867311] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:40.867319] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:40.867329] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=165kB/s, net_out=145kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:40.867350] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399416][2nd=2682]}) [2025-05-31 02:58:40.867616] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=151] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=107, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:40.867720] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=70] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3972, avg_group_log_size=397) [2025-05-31 02:58:40.867741] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3972) [2025-05-31 02:58:40.867755] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:40.869560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.871597] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5977, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:40.871628] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3972) [2025-05-31 02:58:40.871666] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=611, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:40.874442] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=143, avg_log_submit_to_flush_time=6959, avg_log_submit_to_slide_time=7491) [2025-05-31 02:58:40.874503] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=89, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:40.877108] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3972, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:40.877341] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1748678320876816000, start_id=1748678320876816001, end_id=1748678320876816002) [2025-05-31 02:58:40.877662] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:40.880025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.887151] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=59] PNIO [ratelimit] time: 1748678320887147, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007665, add_bytes: 0 [2025-05-31 02:58:40.893209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.905649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.906276] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:40.916117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.926346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.938432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.938691] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678320938687, bytes: 7128358579, bw: 0.153092 MB/s, add_ts: 1002077, add_bytes: 160862 [2025-05-31 02:58:40.944109] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=29] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:58:40.949189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.955912] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-05-31 02:58:40.960435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.964598] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:40.972246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.983518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:40.994053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.004379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.016181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.018762] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A381D-0-0] [lt=54][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525370200885, cache_obj->added_lc()=false, cache_obj->get_object_id()=53597, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:41.019626] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:41.024364] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-05-31 02:58:41.024408] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:41.026706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.037221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.043259] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=27] gc stale ls task succ [2025-05-31 02:58:41.047702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.055414] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=49] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:41.058518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.063269] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:41.063578] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:41.065336] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=566226, clean_num=31457) [2025-05-31 02:58:41.072590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.072791] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [16802][T1_L0_G1][T1][YB42AC10871E-0006366549CC101C-0-0] [lt=847] init point thread id with(&point=0x55c0908cdc40, idx_=418, point=[thread id=16802, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=16802) [2025-05-31 03:58:41.075869] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [17084][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:41.075886] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:58:41.086779] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.097068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.108662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.114666] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA60-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:41.118971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.129258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.139743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.150094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.160044] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=26] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:41.160135] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=77] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:41.160154] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1010) [2025-05-31 02:58:41.160385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.170599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.180900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.191147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.201733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.212022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.220279] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:41.222830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.223453] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=39] === Run print tenant memory usage task === [2025-05-31 02:58:41.223581] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=101] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 121,196,544 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 647,544,832 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,321,222,978 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 52,809,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 895,873,024 max_mem_memstore_can_get_now= 264,046,576 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:58:41.224027] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,073,150,976 total_hold= 3,323,985,920 used= 1,574,588,416 freelists_hold= 1,498,562,560 total_maps= 2,017 total_unmaps= 805 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 121,196,544 virtual_memory_used= 4,344,737,792 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,006,632,960 count= 480 pushes= 14,516,226 pops= 14,515,746 maps= 1,822 unmaps= 761 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,934 pops= 22,930 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:58:41.224080] INFO print (ob_malloc_time_monitor.cpp:37) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=33] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 1176, delta_count= 1, avg_cost_time= 1176 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:58:41.233238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.243398] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.254131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.264497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.267765] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:41.272019] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678321271449000, start_id=1748678321271449000, end_id=1748678321271449001) [2025-05-31 02:58:41.274701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.276284] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=10] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:41.278301] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=31][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.278426] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=118][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.278449] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 08672AE669360600, int: 1748678321268488}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 08672AE669360600, int: 1748678321268488}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.278593] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:45) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=65] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=60645, insert_dup_release_count=60645, col_map_alloc_count=0, col_map_release_count=0) [2025-05-31 02:58:41.278628] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=19][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.283823] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=77] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:41.284016] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83077][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421804} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.284167] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83077][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=111] rpc statistics(total_trans_msg_count=46, total_batch_msg_count=0) [2025-05-31 02:58:41.284179] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:148) [83077][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4337-0-0] [lt=11] transaction rpc statistics(total_rt=5734, total_process=33, avg_rt=168) [2025-05-31 02:58:41.285535] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.285988] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=91] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:41.287924] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=42] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:41.287987] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=61] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:41.290307] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=53] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:41.290400] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=41] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:41.290417] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:41.290443] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:41.291067] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=619] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:41.291109] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=32] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:41.291120] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:41.291143] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:41.291164] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:41.294388] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:41.294434] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=41] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:41.294446] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:41.294485] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:41.294499] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:41.294509] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:41.294521] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:41.294538] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD7-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:41.294612] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137163543}], scn_array=[cnt:1, {val:1748678321278975000, v:0}], curr_size=589, accum_size=3092, time_cost=12413) [2025-05-31 02:58:41.294698] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:41.294718] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5665, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:41.295081] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678320876816001, v:0}, new_max_version={val:1748678320876816001, v:0}, last_min_version={val:1748678320876816000, v:0}, last_max_version={val:1748678320876816000, v:0}, total_time_us=15348, delta_us=418263) [2025-05-31 02:58:41.295107] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678320876816001, v:0}, min_version={val:1748678320876816001, v:0}, max_version={val:1748678320876816001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:41.295991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.306435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.312314] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4338-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1748678321311592000, start_id=1748678321311592000, end_id=1748678321311592001) [2025-05-31 02:58:41.312561] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4338-0-0] [lt=35] transaction rpc statistics(total_rt=6702, total_process=20, avg_rt=319) [2025-05-31 02:58:41.318493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.329057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.332050] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4339-0-0] [lt=10][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.332108] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4339-0-0] [lt=52][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.332169] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4339-0-0] [lt=51][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f4, hex: 8F3C2BE669360600, int: 1748678321323151}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fc, hex: 8F3C2BE669360600, int: 1748678321323151}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c504, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50c, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.332363] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4339-0-0] [lt=96][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.335811] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4339-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10283}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421805} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.342329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.348532] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE433A-0-0] [lt=113] get number(ret=0, service_type_=0, range=1, base_id=1748678321348460000, start_id=1748678321348460000, end_id=1748678321348460001) [2025-05-31 02:58:41.353342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.359265] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.359308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.359335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.359363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.363511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.365371] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=40] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:41.369400] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=43] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.652903938293457031e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.723171724460813493e+03) [2025-05-31 02:58:41.369789] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=61][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.369859] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=65][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.369878] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 11A92BE669360600, int: 1748678321350929}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 11A92BE669360600, int: 1748678321350929}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.370061] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=92][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.371661] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE433B-0-0] [lt=32] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:16185}, to_seq_no={branch:0, seq:8505}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421806} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.374013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.379688] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678321379654], gts:1748678321379027001, latest_srr:[mts=1748678321379654]}) [2025-05-31 02:58:41.379737] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:41.386259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.396952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.411146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.413186] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433D-0-0] [lt=98][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.413221] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433D-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.413239] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433D-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 05672CE669360600, int: 1748678321399557}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 05672CE669360600, int: 1748678321399557}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.413431] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433D-0-0] [lt=113][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.418294] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE433D-0-0] [lt=115] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10937}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421807} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.421095] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:41.422513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=811][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.432969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:41.433634] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [17084][T1_L0_G1][T1][YB42AC1087A7-00063665517F545D-0-0] [lt=28] init point thread id with(&point=0x55c0908d22c0, idx_=700, point=[thread id=17084, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=17084) [2025-05-31 02:58:41.443230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.450424] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433F-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.450474] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433F-0-0] [lt=45][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.450495] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433F-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 6E0D2DE669360600, int: 1748678321442158}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 6E0D2DE669360600, int: 1748678321442158}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.450662] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE433F-0-0] [lt=67][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.452648] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE433F-0-0] [lt=50] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421808} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.453495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.456093] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:41.463978] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.464790] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:41.465463] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83105][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4340-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1748678321465108000, start_id=1748678321465108000, end_id=1748678321465108001) [2025-05-31 02:58:41.466415] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:64K/14:410174 disk:64K/14:410174 t1m2:64K/14:410174 t1m2g0:64K/14:410174 |q0: 0 |q1: 0 [2025-05-31 02:58:41.467488] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1024-0-0] [lt=30] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=6875) [2025-05-31 02:58:41.468534] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:41.476209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.476489] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [14429][T1_L0_G19][T1][YB42AC10871E-00063665511850D6-0-0] [lt=27] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:58:41.476531] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [14429][T1_L0_G19][T1][YB42AC10871E-00063665511850D6-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 03:58:41.477688] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [17207][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:41.477721] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:58:41.485931] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=88][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae9860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421809} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4016} write_epoch=1, prev=this=0x2b09b315d0f0 trans_version={val:1748674846736165000, v:0} scn={val:1748674846736165000, v:0} tx_id={txid:11413758} prev=0x2b09b315cf38 next=(nil) modify_count=1 acc_checksum=143442495 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:34840} write_epoch=3, snapshot_version={val:1748678321476233000, v:0}, *this={this=0x2b09b315cdd8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b09b315d0f0 latest_compact_node=(nil) max_trans_version={val:1748678260583849000, v:0} max_trans_id=11421725 max_elr_trans_version={val:1748678260583849000, v:0} max_elr_trans_id=11421725 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748674846736165000, v:0} min_modify_scn={val:1748674846736165000, v:0}}) [2025-05-31 02:58:41.486009] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=76][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae9860 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421809} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4016} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351378281, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351378281, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321476233000, v:0}, tx_id:{txid:11421809}, scn:{branch:0, seq:4011}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421809}, tx_desc:{this:0x2b094e9f2520, tx_id:{txid:11421809}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321485461, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321497866, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:4016}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.486149] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=137][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421809}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4016}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.486173] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=22][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421809} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.486204] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.486217] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: E27F2DE669360600, int: 1748678321471458}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: E27F2DE669360600, int: 1748678321471458}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 03085B0800000000, int: 140183555}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 00A0B20900000000, int: 162701312}]}) [2025-05-31 02:58:41.486308] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.490870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.491097] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83075][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4341-0-0] [lt=19] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:20252}, to_seq_no={branch:0, seq:4012}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421809} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 03:58:41.491418] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [17212][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 03:58:41.491471] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [17212][T1_L0_G1][T1][YB42AC1087A7-00063665517F545F-0-0] [lt=25] init point thread id with(&point=0x55c0908d42c0, idx_=828, point=[thread id=17212, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=17212) [2025-05-31 02:58:41.493405] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:41.503602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.503976] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4342-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1748678321502347000, start_id=1748678321502347000, end_id=1748678321502347001) [2025-05-31 02:58:41.513925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.517434] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=93][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988ae9ae0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421810} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2b098033a8b0 trans_version={val:1748677894753947000, v:0} scn={val:1748677894753947000, v:0} tx_id={txid:11421229} prev=0x2b097cd36870 next=(nil) modify_count=11 acc_checksum=1694536533 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:48530} write_epoch=0, snapshot_version={val:1748678321513350000, v:0}, *this={this=0x2b09b315d250 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098033a8b0 latest_compact_node=0x2b098033a8b0 max_trans_version={val:1748678260648501000, v:0} max_trans_id=11421726 max_elr_trans_version={val:1748678260648501000, v:0} max_elr_trans_id=11421726 latest_compact_ts=1748677894765269 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677894753947000, v:0} min_modify_scn={val:1748675821756142000, v:0}}) [2025-05-31 02:58:41.517498] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988ae9ae0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421810} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2b091c8d31a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351415543, tablet_id:{id:309}, table_iter:0x2b091c8d16d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351415543, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321513350000, v:0}, tx_id:{txid:11421810}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421810}, tx_desc:{this:0x2b098d901880, tx_id:{txid:11421810}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321516065, commit_ts:-1, finish_ts:-1, timeout_us:29999982, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321543358, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.517644] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=144][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421810}, data:0x2b091c8d00c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.517664] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.517696] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.517711] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098a80c820, local_datums:0x2b098a80c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b098a80c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b098a80c4e2, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c4f2, hex: F21C2EE669360600, int: 1748678321511666}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c4fa, hex: F21C2EE669360600, int: 1748678321511666}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c502, hex: BD69750000000000, int: 7694781}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098a80c50a, hex: 0020B40000000000, int: 11804672}]}) [2025-05-31 02:58:41.517825] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.520086] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4343-0-0] [lt=35] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6024}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421810} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.524654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.536449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.548017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.555923] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4345-0-0] [lt=10][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.555974] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4345-0-0] [lt=46][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.555994] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4345-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f4, hex: 8A9C2EE669360600, int: 1748678321544330}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fc, hex: 8A9C2EE669360600, int: 1748678321544330}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c504, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50c, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.556130] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4345-0-0] [lt=65][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.558278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.559243] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4345-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421811} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.568749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.579028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.584213] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:41.589300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.590886] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=55] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.591135] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.591227] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.591258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.591284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.592125] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.592170] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:41.592724] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4347-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.592798] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4347-0-0] [lt=70][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.592816] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4347-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f4, hex: 392F2FE669360600, int: 1748678321581881}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fc, hex: 392F2FE669360600, int: 1748678321581881}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c504, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50c, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.593040] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4347-0-0] [lt=134][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.599042] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83080][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4347-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5835}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421812} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.599565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.608774] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:164) [81744][T1_TimerWK3_MediumLoop][T1][YB42AC1087A6-00063665431A60B5-0-0] [lt=33] MediumCheckTask(cost_ts=9) [2025-05-31 02:58:41.610333] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.621081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.624080] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:41.631930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.640123] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=15][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aea420 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421814} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:13856} write_epoch=1, prev=this=0x2b09803468b0 trans_version={val:1748678260809040000, v:0} scn={val:1748678260809040000, v:0} tx_id={txid:11421730} prev=0x2b0988adcfa0 next=(nil) modify_count=113 acc_checksum=3821799604 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:43787} write_epoch=0, snapshot_version={val:1748678321635921000, v:0}, *this={this=0x2b09b315db40 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b09803468b0 latest_compact_node=0x2b09803468b0 max_trans_version={val:1748678260809040000, v:0} max_trans_id=11421730 max_elr_trans_version={val:1748678260809040000, v:0} max_elr_trans_id=11421730 latest_compact_ts=1748678260814753 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678260809040000, v:0} min_modify_scn={val:1748674846969018000, v:0}}) [2025-05-31 02:58:41.640213] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=88][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aea420 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421814} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:13856} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351534327, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351534327, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321635921000, v:0}, tx_id:{txid:11421814}, scn:{branch:0, seq:8069}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421814}, tx_desc:{this:0x2b094e9f65e0, tx_id:{txid:11421814}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321638475, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321657779, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:13856}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.640425] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=208][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421814}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:13856}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.640447] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=21][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.640481] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=31][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.640498] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 66EC2FE669360600, int: 1748678321630310}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 66EC2FE669360600, int: 1748678321630310}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 6AC5710D00000000, int: 225559914}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 6AC5710D00000000, int: 225559914}]}) [2025-05-31 02:58:41.640668] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.642401] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3D0-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525370824575, cache_obj->added_lc()=false, cache_obj->get_object_id()=53598, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:41.642764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.645794] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434B-0-0] [lt=85] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:17492}, to_seq_no={branch:0, seq:8070}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421814} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.654375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.662749] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=25] ====== traversal_flush timer task ====== [2025-05-31 02:58:41.663236] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B6-0-0] [lt=42] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678321570553000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678330364653000, latest_log_ts:{val:1748678313702670001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:41.663424] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60B6-0-0] [lt=179] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:41.665171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.666258] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=56926671, this={dio_aligned_buf:{buf_write_offset:1330, buf_padding_size:2766, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:389608, truncate_used_ts:132113}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=56926208, buf_len=867, total_write_size=65267, total_write_size_after_dio=630784, ob_pwrite_used_ts=539070, count=138, each_pwrite_cost=3906) [2025-05-31 02:58:41.671595] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:41.672385] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:41.673537] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F0D-0-0] [lt=39] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=68) [2025-05-31 02:58:41.673754] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1748678321670033000, start_id=1748678321670033001, end_id=1748678321670033002) [2025-05-31 02:58:41.675391] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=67][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aea878 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421815} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:12325} write_epoch=1, prev=this=0x2b0980346980 trans_version={val:1748678260838368000, v:0} scn={val:1748678260838368000, v:0} tx_id={txid:11421731} prev=0x2b0988add3f8 next=(nil) modify_count=113 acc_checksum=2138333129 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:19161} write_epoch=0, snapshot_version={val:1748678321670033000, v:0}, *this={this=0x2b097ccca558 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980346980 latest_compact_node=0x2b0980346980 max_trans_version={val:1748678260838368000, v:0} max_trans_id=11421731 max_elr_trans_version={val:1748678260838368000, v:0} max_elr_trans_id=11421731 latest_compact_ts=1748678260842430 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678260838368000, v:0} min_modify_scn={val:1748674847029540000, v:0}}) [2025-05-31 02:58:41.675467] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=74][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aea878 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421815} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:12325} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351570789, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351570789, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321670033000, v:0}, tx_id:{txid:11421815}, scn:{branch:0, seq:12322}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421815}, tx_desc:{this:0x2b094e99f5c0, tx_id:{txid:11421815}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321674977, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321694734, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:12325}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.675563] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421815}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:12325}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.675576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.675581] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421815} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.675790] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=205][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.675916] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=119][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 5D7730E669360600, int: 1748678321665885}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 5D7730E669360600, int: 1748678321665885}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: F236B90900000000, int: 163133170}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: F236B90900000000, int: 163133170}]}) [2025-05-31 02:58:41.676066] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=106][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.678927] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434D-0-0] [lt=43] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:16266}, to_seq_no={branch:0, seq:12323}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421815} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.688029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.694029] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434E-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678321692271000, start_id=1748678321692271000, end_id=1748678321692271001) [2025-05-31 02:58:41.698342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.705465] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434F-0-0] [lt=46][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.705520] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434F-0-0] [lt=50][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.705592] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434F-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099ee0c820, local_datums:0x2b099ee0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b099ee0c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b099ee0c4e2, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4f3, hex: 26F230E669360600, int: 1748678321697318}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4fb, hex: 26F230E669360600, int: 1748678321697318}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.705784] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83073][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434F-0-0] [lt=90][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.709071] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.709294] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE434F-0-0] [lt=42] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10081}, to_seq_no={branch:0, seq:2497}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421816} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.718971] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=57] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:41.719015] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] temporary file module use no blocks [2025-05-31 02:58:41.719026] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:41.719042] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:41.719150] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=105] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525369900831, flush_io_finished_ret_:0, flush_io_finished_round_:19889, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:41.719664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.725324] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83090][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4350-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1748678321724741000, start_id=1748678321724741000, end_id=1748678321724741001) [2025-05-31 02:58:41.730056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.737906] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4351-0-0] [lt=33][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.737987] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4351-0-0] [lt=34][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.738011] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4351-0-0] [lt=15][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 036831E669360600, int: 1748678321727491}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 036831E669360600, int: 1748678321727491}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.738267] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4351-0-0] [lt=155][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.743247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.743672] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4351-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:12630}, to_seq_no={branch:0, seq:2364}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421817} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.750430] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=13] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:41.750477] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=42] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:41.750493] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:41.750504] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:41.753471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.753357] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:41.754240] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=879] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:41.758487] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=92] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:41.764064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.770431] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4472-0-0] [lt=72][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:41.774313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.781456] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=30][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aeaff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421818} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:14502} write_epoch=1, prev=this=0x2b09b315e610 trans_version={val:1748674847205046000, v:0} scn={val:1748674847205046000, v:0} tx_id={txid:11413787} prev=0x2b09802c7230 next=(nil) modify_count=1 acc_checksum=3655408859 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:28743} write_epoch=3, snapshot_version={val:1748678321777201000, v:0}, *this={this=0x2b097ccca748 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b09b315e610 latest_compact_node=(nil) max_trans_version={val:1748678260946246000, v:0} max_trans_id=11421734 max_elr_trans_version={val:1748678260946246000, v:0} max_elr_trans_id=11421734 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748674847205046000, v:0} min_modify_scn={val:1748674847205046000, v:0}}) [2025-05-31 02:58:41.781522] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aeaff8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421818} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:14502} write_epoch=1, ctx={this:0x2b091d6551a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351676747, tablet_id:{id:309}, table_iter:0x2b091d6536d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351676747, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321777201000, v:0}, tx_id:{txid:11421818}, scn:{branch:0, seq:7505}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421818}, tx_desc:{this:0x2b09899fc700, tx_id:{txid:11421818}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321781022, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321797336, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:14502}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.781676] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=151][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421818}, data:0x2b091d6520c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:14502}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.781695] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421818} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.781727] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.781741] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099880c820, local_datums:0x2b099880c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b099880c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b099880c4e2, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4f3, hex: 541B32E669360600, int: 1748678321773396}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c4fb, hex: 541B32E669360600, int: 1748678321773396}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c503, hex: 00606F0A00000000, int: 175071232}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099880c50b, hex: 00606F0A00000000, int: 175071232}]}) [2025-05-31 02:58:41.781937] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.784478] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4353-0-0] [lt=62] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14505}, to_seq_no={branch:0, seq:7506}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421818} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.785361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.798102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.808450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.820066] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=99][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aeb278 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421819} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=1, prev=this=0x2b098033aca0 trans_version={val:1748677895270740000, v:0} scn={val:1748677895270740000, v:0} tx_id={txid:11421238} prev=0x2b097cd36c10 next=(nil) modify_count=11 acc_checksum=1717750990 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:28962} write_epoch=0, snapshot_version={val:1748678321812157000, v:0}, *this={this=0x2b09802c7378 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098033aca0 latest_compact_node=0x2b098033aca0 max_trans_version={val:1748678260996120000, v:0} max_trans_id=11421735 max_elr_trans_version={val:1748678260996120000, v:0} max_elr_trans_id=11421735 latest_compact_ts=1748677895282576 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677895270740000, v:0} min_modify_scn={val:1748675822147143000, v:0}}) [2025-05-31 02:58:41.820141] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=72][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aeb278 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421819} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:93} seq_no={branch:0, seq:7} write_epoch=1, ctx={this:0x2b091c8d31a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351712639, tablet_id:{id:309}, table_iter:0x2b091c8d16d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351712639, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321812157000, v:0}, tx_id:{txid:11421819}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421819}, tx_desc:{this:0x2b098d8dd1c0, tx_id:{txid:11421819}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321819375, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321841295, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:7}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.820286] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=142][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421819}, data:0x2b091c8d00c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.820305] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421819} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.820337] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.820352] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aca0c820, local_datums:0x2b09aca0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aca0c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09aca0c4e2, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4f3, hex: FF9E32E669360600, int: 1748678321807103}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4fb, hex: FF9E32E669360600, int: 1748678321807103}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c503, hex: 00B03E0100000000, int: 20885504}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c50b, hex: 00B03E0100000000, int: 20885504}]}) [2025-05-31 02:58:41.820502] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.820686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.822833] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4355-0-0] [lt=46] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421819} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.827120] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=97] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:41.831104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.841402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.852190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.862636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.866375] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4359-0-0] [lt=15][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:41.866466] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4359-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.866489] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4359-0-0] [lt=14][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: 5D7333E669360600, int: 1748678321861469}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: 5D7333E669360600, int: 1748678321861469}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:41.866829] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4359-0-0] [lt=226][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.872418] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83099][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4359-0-0] [lt=47] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10742}, to_seq_no={branch:0, seq:4940}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421821} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.873378] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:41.873457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.874325] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:41.874407] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:26, norm_iops:7, bw:121970, [delay/us]:prepare:17, schedule:401, submit:235, rt:2674, total:3329) [2025-05-31 02:58:41.874426] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=7, iops=7, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:41.874513] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:26, avg_bw:121967, [delay/us]: prepare:17, schedule:401, submit:235, device:2674, total:3329) [2025-05-31 02:58:41.874529] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:41.874555] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4065, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:674, total:688) [2025-05-31 02:58:41.874608] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:41.874665] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4065, [delay/us]: prepare:0, schedule:0, submit:600000000, device:674, total:688) [2025-05-31 02:58:41.874686] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:41.874702] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:41.874710] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:41.874719] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:41.874727] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:41.874736] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=606kB/s, net_out=209kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:41.874756] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399417][2nd=2964]}) [2025-05-31 02:58:41.884938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.887980] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=26, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:41.888020] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=26, avg_log_batch_cnt=1, total_group_log_size=15649, avg_group_log_size=601) [2025-05-31 02:58:41.888040] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=15182) [2025-05-31 02:58:41.888055] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=26, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:41.888715] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83075][T1_L0_G0][T1][YB42AC1087A8-000636672FFE435A-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678321888138000, start_id=1748678321888138000, end_id=1748678321888138001) [2025-05-31 02:58:41.890211] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT WRITE LOG COST TIME](cur_stat_count=26, stat_interval=1000000, avg (count/cost)=3395, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:41.890238] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=15649) [2025-05-31 02:58:41.890304] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=26, stat_interval=1000000, avg (count/cost)=523, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:41.890490] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=26, stat_interval=1000000, avg (count/cost)=39, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:41.891309] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1748678321891305, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004158, add_bytes: 0 [2025-05-31 02:58:41.895454] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.897092] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [16802][T1_L0_G1][T1][YB42AC1087A7-00063665517F546E-0-0] [lt=35] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=26, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=108, avg_log_submit_to_flush_time=4319, avg_log_submit_to_slide_time=5293) [2025-05-31 02:58:41.906073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.907227] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=77] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=15649, hit_cnt=26, read_cnt=26, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:41.907438] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:41.916386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.917259] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=71][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aeb890 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421822} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4120} write_epoch=1, prev=this=0x2b09b317c3f0 trans_version={val:1748674969625020000, v:0} scn={val:1748674969625020000, v:0} tx_id={txid:11417252} prev=0x2b09b317c238 next=(nil) modify_count=3 acc_checksum=2995272462 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:50336} write_epoch=3, snapshot_version={val:1748678321913244000, v:0}, *this={this=0x2b09b315ebd8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b09b317c3f0 latest_compact_node=(nil) max_trans_version={val:1748678261075134000, v:0} max_trans_id=11421738 max_elr_trans_version={val:1748678261075134000, v:0} max_elr_trans_id=11421738 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748674969625020000, v:0} min_modify_scn={val:1748674847389205000, v:0}}) [2025-05-31 02:58:41.917374] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=112][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aeb890 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421822} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4120} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351814165, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351814165, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321913244000, v:0}, tx_id:{txid:11421822}, scn:{branch:0, seq:4117}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421822}, tx_desc:{this:0x2b094e9d5fe0, tx_id:{txid:11421822}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321914838, commit_ts:-1, finish_ts:-1, timeout_us:29999977, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321938829, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:4120}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.917475] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421822}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4120}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.917493] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421822} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.917525] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.917541] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: D33A34E669360600, int: 1748678321912531}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: D33A34E669360600, int: 1748678321912531}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0060090000000000, int: 614400}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0060090000000000, int: 614400}]}) [2025-05-31 02:58:41.917698] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.920924] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE435B-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10475}, to_seq_no={branch:0, seq:4118}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421822} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.927462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.937972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.942672] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aebb10 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421823} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7916} write_epoch=1, prev=this=0x2b0980346cf0 trans_version={val:1748678261116942000, v:0} scn={val:1748678261116942000, v:0} tx_id={txid:11421739} prev=0x2b0988ade738 next=(nil) modify_count=113 acc_checksum=405826781 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:26737} write_epoch=0, snapshot_version={val:1748678321940864000, v:0}, *this={this=0x2b09b315f050 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980346cf0 latest_compact_node=0x2b0980346cf0 max_trans_version={val:1748678261116942000, v:0} max_trans_id=11421739 max_elr_trans_version={val:1748678261116942000, v:0} max_elr_trans_id=11421739 latest_compact_ts=1748678261125125 last_compact_cnt=113 total_trans_node_cnt=114 max_modify_scn={val:1748678261116942000, v:0} min_modify_scn={val:1748674847443094000, v:0}}) [2025-05-31 02:58:41.942737] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aebb10 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421823} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7916} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351839815, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351839815, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321940864000, v:0}, tx_id:{txid:11421823}, scn:{branch:0, seq:4}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421823}, tx_desc:{this:0x2b094e85fa80, tx_id:{txid:11421823}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321941936, commit_ts:-1, finish_ts:-1, timeout_us:29999927, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678321966268, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:7916}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.942835] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421823}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7916}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.942854] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421823} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.942965] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=107][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.942982] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: C49034E669360600, int: 1748678321934532}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: C49034E669360600, int: 1748678321934532}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 0050370000000000, int: 3624960}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 0050370000000000, int: 3624960}]}) [2025-05-31 02:58:41.943127] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=87][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.944444] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE435D-0-0] [lt=86] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7919}, to_seq_no={branch:0, seq:5}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421823} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.944565] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1748678321944562, bytes: 7129026756, bw: 0.633501 MB/s, add_ts: 1005875, add_bytes: 668177 [2025-05-31 02:58:41.949488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.956348] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-31 02:58:41.960075] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.968183] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:41.971594] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.984400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:41.990413] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aec130 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421825} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:10357} write_epoch=1, prev=this=0x2b09b315f7d8 trans_version={val:1748674847518208000, v:0} scn={val:1748674847518208000, v:0} tx_id={txid:11413813} prev=0x2b09b315f620 next=(nil) modify_count=1 acc_checksum=2503410586 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:38936} write_epoch=3, snapshot_version={val:1748678321979508000, v:0}, *this={this=0x2b09b315f4c0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b09b315f7d8 latest_compact_node=(nil) max_trans_version={val:1748678261174468000, v:0} max_trans_id=11421741 max_elr_trans_version={val:1748678261174468000, v:0} max_elr_trans_id=11421741 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748674847518208000, v:0} min_modify_scn={val:1748674847518208000, v:0}}) [2025-05-31 02:58:41.990477] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=61][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aec130 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421825} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:10357} write_epoch=1, ctx={this:0x2b091c8d31a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351881844, tablet_id:{id:309}, table_iter:0x2b091c8d16d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351881844, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678321979508000, v:0}, tx_id:{txid:11421825}, scn:{branch:0, seq:2537}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421825}, tx_desc:{this:0x2b098d8b6aa0, tx_id:{txid:11421825}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678321990156, commit_ts:-1, finish_ts:-1, timeout_us:29999968, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678322007861, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:10357}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:41.990572] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421825}, data:0x2b091c8d00c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:10357}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:41.990590] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421825} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:41.990620] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:41.990635] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aca0c820, local_datums:0x2b09aca0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aca0c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09aca0c4e2, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4f2, hex: 253835E669360600, int: 1748678321977381}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c4fa, hex: 253835E669360600, int: 1748678321977381}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c502, hex: 00C0BC0000000000, int: 12369920}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aca0c50a, hex: 00C0BC0000000000, int: 12369920}]}) [2025-05-31 02:58:41.990741] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:41.994219] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4361-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10360}, to_seq_no={branch:0, seq:2538}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421825} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:41.994828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.005353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.015701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.026686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=164][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.027702] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:42.028397] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aec3b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421826} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1383} write_epoch=1, prev=this=0x2b0988a8c558 trans_version={val:1748677895598998000, v:0} scn={val:1748677895598998000, v:0} tx_id={txid:11421245} prev=0x2b0988a8c3a0 next=(nil) modify_count=9 acc_checksum=647961562 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678321076063000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:26319} write_epoch=3, snapshot_version={val:1748678322020996000, v:0}, *this={this=0x2b09b315f938 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b0988a8c558 latest_compact_node=0x2b0980318f80 max_trans_version={val:1748678261206291000, v:0} max_trans_id=11421742 max_elr_trans_version={val:1748678261206291000, v:0} max_elr_trans_id=11421742 latest_compact_ts=1748677041603237 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677895598998000, v:0} min_modify_scn={val:1748675822366852001, v:0}}) [2025-05-31 02:58:42.028459] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=59][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aec3b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11421826} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:1383} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678351926033, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678351926033, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678322020996000, v:0}, tx_id:{txid:11421826}, scn:{branch:0, seq:1380}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11421826}, tx_desc:{this:0x2b094e81ce20, tx_id:{txid:11421826}, state:1, addr:"172.16.135.168:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678322020996, commit_ts:-1, finish_ts:-1, timeout_us:29999979, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678322049523, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:1383}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:58:42.028557] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11421826}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:1383}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:58:42.028575] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.168", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":12},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11421826} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:58:42.028605] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:42.028663] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f2, hex: CADC35E669360600, int: 1748678322019530}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4fa, hex: CADC35E669360600, int: 1748678322019530}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c502, hex: 00908A0000000000, int: 9080832}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c50a, hex: 00908A0000000000, int: 9080832}]}) [2025-05-31 02:58:42.028776] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=63][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:42.028951] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A381F-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525371211120, cache_obj->added_lc()=false, cache_obj->get_object_id()=53599, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:42.032985] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4363-0-0] [lt=111] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8590}, to_seq_no={branch:0, seq:1381}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421826} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:42.037194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.041194] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=33] current monitor number(seq_=-1) [2025-05-31 02:58:42.043711] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=28] gc stale ls task succ [2025-05-31 02:58:42.047224] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83095][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4364-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1748678322046394000, start_id=1748678322046394000, end_id=1748678322046394001) [2025-05-31 02:58:42.047752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.055882] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:42.056151] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83092][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33B7-0-0] [lt=62][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525371238321, cache_obj->added_lc()=false, cache_obj->get_object_id()=53600, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:42.058522] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137179828}], scn_array=[cnt:1, {val:1748678322046394000, v:0}], curr_size=563, accum_size=3702, time_cost=10301) [2025-05-31 02:58:42.058802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.063366] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2062-0-0] [lt=141] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:42.063402] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2062-0-0] [lt=25] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=37) [2025-05-31 02:58:42.069708] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4365-0-0] [lt=11][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:58:42.069748] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4365-0-0] [lt=36][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:58:42.069767] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4365-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c620c820, local_datums:0x2b09c620c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09c620c4c4, hex: 3137322E31362E3133352E313638, cstr: 172.16.135.168}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4da, hex: 0C00000000000000, int: 12}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c620c4e2, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f1, hex: 6C8B36E669360600, int: 1748678322064236}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c4f9, hex: 6C8B36E669360600, int: 1748678322064236}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c620c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:58:42.069995] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672FFE4365-0-0] [lt=68][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:58:42.072093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.072998] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:42.074527] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:42.074801] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC1087A8-000636672FFE4365-0-0] [lt=43] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:16530}, to_seq_no={branch:0, seq:8558}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11421827} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:58:42.082316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.092571] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.103395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.115505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.130945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.137978] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA61-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:42.142261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.153601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.165511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.176135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.188516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.199410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.205110] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:48) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=54] send to rs all task status succeed [2025-05-31 02:58:42.209823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.214017] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:42.220145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.230632] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:42.230948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.241679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.251968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.263109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.273660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.276528] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4179-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678322275486000, start_id=1748678322275486000, end_id=1748678322275486001) [2025-05-31 02:58:42.277758] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:42.284252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.285390] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=44] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:42.285410] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=20] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:42.285463] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3B-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:42.285484] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3B-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:42.294960] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=34] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:42.296089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.296528] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=36] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:42.296702] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=94] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:42.301917] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=92] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=33, stat_interval=1000000, avg (count/cost)=17, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:42.301963] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT LOG LIFE TIME](cur_stat_count=33, stat_interval=1000000, avg (count/cost)=5662, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:42.302607] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678321973188000, v:0}, new_max_version={val:1748678321973188000, v:0}, last_min_version={val:1748678321832666000, v:0}, last_max_version={val:1748678321832666000, v:0}, total_time_us=5600, delta_us=329416) [2025-05-31 02:58:42.302658] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678321973188000, v:0}, min_version={val:1748678321973188000, v:0}, max_version={val:1748678321973188000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:42.308635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.320229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.331203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.341452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.351720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.359472] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.359536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.359567] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.359594] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.362074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.373973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.376173] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3821-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525371558342, cache_obj->added_lc()=false, cache_obj->get_object_id()=53601, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:42.378246] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3823-0-0] [lt=26][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:42.379216] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3825-0-0] [lt=25][errcode=-4201] hashtable not init [2025-05-31 02:58:42.379496] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=107] get number(ret=0, service_type_=0, range=1, base_id=1748678322379383000, start_id=1748678322379383000, end_id=1748678322379383001) [2025-05-31 02:58:42.381669] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ts response task statistics(alloc_count=485882, free_count=242940) [2025-05-31 02:58:42.381766] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=87] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:42.381826] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678321880492], gts:1748678321913929000, latest_srr:[mts=1748678322381782]}) [2025-05-31 02:58:42.386708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.397372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.408254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.420112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.430369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.431435] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:42.440927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.453211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.456574] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-31 02:58:42.464077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.469079] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=74] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:42.476263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.476324] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:58:42.478268] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:42.485256] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [14429][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=39] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:58:42.486537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.496992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.497153] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:42.497350] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [14466][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=47] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:42.502114] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF94C-0-0] [lt=83] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:42.503256] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=34] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:132K/29:277554 disk:132K/29:277554 t1m2:132K/29:277554 t1m2g0:132K/29:277554 |q0: 0 |q1: 0 [2025-05-31 02:58:42.505227] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1042-0-0] [lt=45] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=17769) [2025-05-31 02:58:42.508042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.509431] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] get_max_applied_scn(scn={val:1748678322379383000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137184737}, palf_committed_end_scn_:{val:1748678322502059000, v:0}, last_check_scn_:{val:1748678322502059000, v:0}, max_applied_cb_scn_:{val:1748678322379383000, v:0}}) [2025-05-31 02:58:42.509523] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137184737}, readable_log_tail:{lsn:2137184737}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10167839, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:42.522969] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C7-0-0] [lt=41] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:42.525119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.537818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.545795] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81744][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=33] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:42.548318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.558947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.569141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.576937] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81065][TimerWK3_ServerTracerTim][T0][YB42AC1087A6-0006366545EA56B8-0-0] [lt=12] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:42.581647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.590532] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A67-0-0] [lt=30] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:42.591346] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.591723] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.591759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.591786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.591808] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.591913] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.592044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.592397] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:42.601241] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=171] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:42.602456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.612730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.622934] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.631680] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:42.633415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.646001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.656453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.659745] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=59) [2025-05-31 02:58:42.666748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.677780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.679177] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:42.688196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=203][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.700090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=324][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.710569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.721155] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:42.721194] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-05-31 02:58:42.721205] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:42.721221] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:42.721235] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525370901366, flush_io_finished_ret_:0, flush_io_finished_round_:19890, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:42.721388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.731944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.744519] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.747422] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=35] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:42.747472] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=47] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:42.748651] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] update_check_range(min_check_version=1981, max_check_version=1982, global_version=1983) [2025-05-31 02:58:42.749449] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [17207][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AC-0-0] [lt=39] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:58:42.749528] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [17207][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AC-0-0] [lt=64] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:58:42.756666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.773606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.775719] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4473-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:42.776183] INFO [STORAGE] close_and_rename (ob_storage_file.cpp:1073) [84310][IO_SYNC_CH0][T0][YB42AC1087A8-000636672F5E0F91-0-0] [lt=15] succeed to rename file after close(path_=/nfs_data/ob/archive/check_file/1004_connect_file_20250531T035842.obbak.tmp.1748678322745642, real_path_=/nfs_data/ob/archive/check_file/1004_connect_file_20250531T035842.obbak) [2025-05-31 02:58:42.776255] WDIAG [COMMON] finish (ob_io_define.cpp:900) [84310][IO_SYNC_CH0][T0][YB42AC1087A8-000636672F5E0F91-0-0] [lt=61][errcode=0] invalid storage id(storage_info={storage_id:18446744073709551615, storage_used_mod:2}) [2025-05-31 02:58:42.784941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.786953] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 152 276 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 52 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 16 1 2 0 1 18 6 3 0 1 20 2 1 0 500 0 88 262 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 1 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 38336 [2025-05-31 02:58:42.787023] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 897,970,176 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 453,353,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 109,051,904 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 12,603,392 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:42.787403] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 453,353,472 used= 342,835,344 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 31 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,494,144 used= 31,433,536 count= 8 avg_used= 3,929,192 block_cnt= 8 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 18,317,312 used= 18,097,728 count= 31 avg_used= 583,797 block_cnt= 31 chunk_cnt= 19 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,274,496 used= 6,389,760 count= 108 avg_used= 59,164 block_cnt= 108 chunk_cnt= 28 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,818,048 used= 2,380,480 count= 86 avg_used= 27,680 block_cnt= 86 chunk_cnt= 26 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,015,808 used= 498,976 count= 62 avg_used= 8,048 block_cnt= 62 chunk_cnt= 21 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 425,728 used= 409,768 count= 7 avg_used= 58,538 block_cnt= 7 chunk_cnt= 6 mod=CommSysVarFac [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 412,032 used= 395,328 count= 87 avg_used= 4,544 block_cnt= 87 chunk_cnt= 25 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 253,952 used= 245,760 count= 1 avg_used= 245,760 block_cnt= 1 chunk_cnt= 1 mod=SqlExecutor [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 100,784 used= 26,128 count= 370 avg_used= 70 block_cnt= 99 chunk_cnt= 25 mod=Coro [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 57,024 used= 44,352 count= 66 avg_used= 672 block_cnt= 61 chunk_cnt= 24 mod=[T]ObWarningBuf [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 48,000 used= 33,600 count= 75 avg_used= 448 block_cnt= 45 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 45,264 used= 4,824 count= 201 avg_used= 24 block_cnt= 75 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 3 mod=SqlSession [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PlanVaIdx [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSessiVarMap [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,456 used= 2,304 count= 6 avg_used= 384 block_cnt= 6 chunk_cnt= 4 mod=TableLockNode [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=SequenceIdMap [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,408 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=PackStateMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=ContextsMap [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 342,835,344 used= 334,746,035 count= 32,617 avg_used= 10,262 mod=SUMMARY [2025-05-31 02:58:42.787493] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=73] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:58:42.787515] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:58:42.787534] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,574,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,511,808 used= 1,476,096 count= 186 avg_used= 7,936 block_cnt= 186 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,360,000 used= 1,240,000 count= 625 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,574,720 used= 35,241,449 count= 3,357 avg_used= 10,497 mod=SUMMARY [2025-05-31 02:58:42.787551] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:58:42.787564] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 109,051,904 used= 105,283,584 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 block_cnt= 204 chunk_cnt= 52 mod=CoStack [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:58:42.787583] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:58:42.787628] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:58:42.787657] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:42.787675] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:42.787717] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=31] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:58:42.787734] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 8,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,160 used= 3,968 count= 1 avg_used= 3,968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_ADMIN_SET [MEMORY] hold= 4,160 used= 3,968 count= 1 avg_used= 3,968 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_REMOTE_SY [MEMORY] hold= 8,320 used= 7,936 count= 2 avg_used= 3,968 mod=SUMMARY [2025-05-31 02:58:42.787755] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 12,603,392 used= 8,961,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,770,880 used= 5,634,560 count= 710 avg_used= 7,936 block_cnt= 710 chunk_cnt= 4 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 8,961,536 used= 8,796,224 count= 714 avg_used= 12,319 mod=SUMMARY [2025-05-31 02:58:42.787771] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:42.787807] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 647,544,832 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:42.788262] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 181,188,784 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT start_service_time FROM __all_server WHERE svr_ip = ? AND svr_port = ?) [2025-05-31 02:58:42.796146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.806713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.817271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.828147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.831932] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:42.839087] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.844836] INFO [STORAGE] close_and_rename (ob_storage_file.cpp:1073) [84310][IO_SYNC_CH0][T0][YB42AC1087A8-000636672F5E0F91-0-0] [lt=13] succeed to rename file after close(path_=/nfs_data/ob/archive/format.obbak.tmp.1748678322825567, real_path_=/nfs_data/ob/archive/format.obbak) [2025-05-31 02:58:42.844938] WDIAG [COMMON] finish (ob_io_define.cpp:900) [84310][IO_SYNC_CH0][T0][YB42AC1087A8-000636672F5E0F91-0-0] [lt=30][errcode=0] invalid storage id(storage_info={storage_id:18446744073709551615, storage_used_mod:2}) [2025-05-31 02:58:42.850400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.861380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.862087] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-05-31 02:58:42.862117] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=23] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:58:42.862140] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:42.871589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.872486] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=45] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:42.880260] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:42.882173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.885448] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=65] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678322382101], gts:1748678322416343000, latest_srr:[mts=1748678322885425]}) [2025-05-31 02:58:42.894797] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1748678322890755000, start_id=1748678322890755000, end_id=1748678322890755001) [2025-05-31 02:58:42.895164] INFO [LIB] stat (utility.h:1211) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=110] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:42.895205] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9926, avg_group_log_size=551) [2025-05-31 02:58:42.895225] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=9470) [2025-05-31 02:58:42.895238] INFO [LIB] stat (utility.h:1211) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:42.895381] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:42.895430] WDIAG [COMMON] get_iops_scale (ob_io_calibration.cpp:748) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] invalid size(mode=1, size=0, iops_scale=1.000000000000000000e+00) [2025-05-31 02:58:42.895456] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP](tenant_id=1, io_status=group_id:0, group_name:OTHER_GROUPS, mode:REMOTE WRITE, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:126, iops:2, norm_iops:0, bw:247, limit:9223372036854775807, [delay/us]:prepare:11, schedule:271, submit:37, rt:24656, total:24976) [2025-05-31 02:58:42.895536] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:12288 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:18, norm_iops:4, bw:80231, [delay/us]:prepare:18, schedule:244, submit:433, rt:3247, total:3943) [2025-05-31 02:58:42.895552] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=247, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:42.895614] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=55] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:OTHER_GROUPS, mode:REMOTE WRITE, avg_size:127, avg_iops:2, avg_bw:247, [delay/us]: prepare:11, schedule:271, submit:37, device:24656, total:24976) [2025-05-31 02:58:42.895731] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=105] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:18, avg_bw:80217, [delay/us]: prepare:18, schedule:244, submit:433, device:3247, total:3943) [2025-05-31 02:58:42.895743] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:42.895768] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4010, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:562, total:578) [2025-05-31 02:58:42.895953] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=152] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:42.895991] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4010, [delay/us]: prepare:0, schedule:0, submit:600000000, device:562, total:578) [2025-05-31 02:58:42.896011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:42.896025] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:42.896034] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:42.896042] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:42.896050] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:42.896059] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=329kB/s, net_out=154kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:42.896079] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399418][2nd=2259]}) [2025-05-31 02:58:42.896269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.896624] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4323, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:42.896650] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=9926) [2025-05-31 02:58:42.896671] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=459, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:42.896704] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=62, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:42.898750] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678322898746, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007441, add_bytes: 0 [2025-05-31 02:58:42.899554] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1046-0-0] [lt=51] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=99, avg_log_gen_to_submit_time=201, avg_log_submit_to_flush_time=5263, avg_log_submit_to_slide_time=6021) [2025-05-31 02:58:42.899652] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9926, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:42.904098] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=65, stat_interval=5000000, avg (count/cost)=149, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:42.904126] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=27] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=65, stat_interval=5000000, avg (count/cost)=6636, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:42.904135] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=65, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:42.904142] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=65, stat_interval=5000000, avg (count/cost)=-1272, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:42.904149] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=65, stat_interval=5000000, avg (count/cost)=5513, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:42.908197] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=61] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:42.908604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.917706] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=6] rpc statistics(total_trans_msg_count=45, total_batch_msg_count=0) [2025-05-31 02:58:42.918958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:42.921085] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [17667][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:42.921161] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:42.924597] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=51] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:42.932954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.946577] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1748678322946572, bytes: 7129340162, bw: 0.298288 MB/s, add_ts: 1002010, add_bytes: 313406 [2025-05-31 02:58:42.947102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.950735] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=53] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:42.958289] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:42.959647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.960962] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=26] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:58:42.963422] INFO [SQL] alloc_value (ob_sql_session_mgr.cpp:260) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665433E0720-0-0] [lt=36] alloc_session_count(alloc_total_count=2870000) [2025-05-31 02:58:42.969830] INFO [SQL] free_value (ob_sql_session_mgr.cpp:313) [81742][T1_TimerWK1_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=36] free_session_count(free_total_count=2870000) [2025-05-31 02:58:42.970400] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:42.970456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.982196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.992419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:42.993436] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9A-0-0] [lt=38] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678323025854, keepalive_ts:-1}) [2025-05-31 02:58:42.993494] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9A-0-0] [lt=56] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678323025854, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:43.004026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.015003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.024613] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=33] ====== check clog disk timer task ====== [2025-05-31 02:58:43.024646] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:43.027034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.036547] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3827-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525372218716, cache_obj->added_lc()=false, cache_obj->get_object_id()=53602, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.038425] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=188736, replace_num=15728) [2025-05-31 02:58:43.038459] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:43.038533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.041415] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-05-31 02:58:43.043753] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:58:43.048687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.056080] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:43.059298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.072873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.074252] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:43.081616] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:43.083214] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=597683, clean_num=31457) [2025-05-31 02:58:43.084940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=289][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.095002] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [17667][T1_L0_G1][T1][YB42AC1087A7-00063665517F5484-0-0] [lt=52] init point thread id with(&point=0x55c0908db480, idx_=1283, point=[thread id=17667, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=17667) [2025-05-31 02:58:43.095441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.105743] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AE2-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:43.106073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.116674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.129712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.140077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.150626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.159197] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA62-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:43.161026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.173440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.183845] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.198089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=262][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.208341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.218989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.230296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.238629] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:43.243355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.254358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.264933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.277285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.286445] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:43.287785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.299227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.309550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.316588] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC104D-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=26, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:43.316645] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC104D-0-0] [lt=56] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=5578, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:43.317500] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678322921298000, v:0}, new_max_version={val:1748678322921298000, v:0}, last_min_version={val:1748678322908999000, v:0}, last_max_version={val:1748678322908999000, v:0}, total_time_us=9565, delta_us=396199) [2025-05-31 02:58:43.317556] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678322921298000, v:0}, min_version={val:1748678322921298000, v:0}, max_version={val:1748678322921298000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:43.321623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.331929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.344662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.354903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.359573] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.359699] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.359730] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.359755] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.365193] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.375388] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83095][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3829-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525372557557, cache_obj->added_lc()=false, cache_obj->get_object_id()=53603, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.378279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.386017] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:43.389613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.399812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.411552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.421823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.432325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=161][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.441608] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:43.443905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.454120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.458524] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=51] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:43.464596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.473184] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=77] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:43.475239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.487763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.488485] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:43.497974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.508332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.510766] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF95B-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:43.511992] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:60K/15:561275 net_out:253/2:256277 disk:60K/13:608198 t1m1:253/2:256277 t1m2:60K/13:608198 t1m1g0:253/2:256277 t1m2g0:60K/13:608198 |q0: 0 0 |q1: 0 0 [2025-05-31 02:58:43.516916] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC104E-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5962) [2025-05-31 02:58:43.518809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.531595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=148][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.542056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.552306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.554744] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A25-0-0] [lt=39][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525372736914, cache_obj->added_lc()=false, cache_obj->get_object_id()=53604, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.564852] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.576479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.586962] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.593446] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=74] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593502] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593532] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593608] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.593633] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:43.595549] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:43.595646] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.595817] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.595922] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:43.597339] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.601531] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12] start to print blacklist info [2025-05-31 02:58:43.602201] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=44] ls blacklist refresh finish(cost_time=33543) [2025-05-31 02:58:43.608645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.619103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.629536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=156][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.639831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.642452] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:43.650081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.661647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.668977] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=13] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:43.669025] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=28] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=985) [2025-05-31 02:58:43.671870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.684181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.689681] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:43.694556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.704795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.711901] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62] submit log success(service_type_=0, last_id=1748678340364653000, limited_id=1748678340364653000) [2025-05-31 02:58:43.711947] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] submit log with lock success(service_type_=0, last_id=1748678340364653000, limited_id=1748678340364653000) [2025-05-31 02:58:43.716619] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=56] update id meta(ret=0, service_type=0, limited_id=1748678340364653000, latest_scn={val:1748678323711104001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:58:43.716695] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=74] handle submit callback(service_type_=0, success=true, used_ts=4796, log_ts={val:1748678323711104001, v:0}, limited_id=1748678340364653000) [2025-05-31 02:58:43.716711] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:58:43.716721] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:58:43.717574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.718673] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3826-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678323718208000, start_id=1748678323718208000, end_id=1748678323718208001) [2025-05-31 02:58:43.719228] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:43.721951] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:43.721987] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] temporary file module use no blocks [2025-05-31 02:58:43.721998] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:43.722013] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:43.722027] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525371903448, flush_io_finished_ret_:0, flush_io_finished_round_:19891, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:43.727812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.738179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=176][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.749494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.760129] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.770394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.778712] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4474-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:43.783761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.794669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.805758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.815931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.826153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.836538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.845190] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:43.846825] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.857401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.867691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.880070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.890444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.891277] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:43.900765] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=92, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:43.900787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.900816] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5046, avg_group_log_size=420) [2025-05-31 02:58:43.900839] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5969) [2025-05-31 02:58:43.900965] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=121] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:43.904303] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:43.904363] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:13, norm_iops:3, bw:60897, [delay/us]:prepare:18, schedule:685, submit:192, rt:2043, total:2939) [2025-05-31 02:58:43.904382] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=3, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:43.904443] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:13, avg_bw:60909, [delay/us]: prepare:18, schedule:685, submit:192, device:2043, total:2939) [2025-05-31 02:58:43.904510] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:43.904544] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4060, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:237, total:273) [2025-05-31 02:58:43.904571] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:43.904666] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=89] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4060, [delay/us]: prepare:0, schedule:0, submit:600000000, device:237, total:273) [2025-05-31 02:58:43.904747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:43.904768] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:43.904777] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:43.904786] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:43.904794] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:43.904803] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=182kB/s, net_out=149kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:43.904823] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399419][2nd=2541]}) [2025-05-31 02:58:43.905237] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3349, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:43.905268] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5046) [2025-05-31 02:58:43.905307] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=378, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:43.905707] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=150, avg_log_gen_to_submit_time=265, avg_log_submit_to_flush_time=3804, avg_log_submit_to_slide_time=5336) [2025-05-31 02:58:43.905762] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=46, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:43.906363] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=471] PNIO [ratelimit] time: 1748678323906360, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-05-31 02:58:43.906778] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5046, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:43.913297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.913698] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=54] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:43.926954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.927010] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:43.927595] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [17212][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=50] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:43.929569] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=76] XA scheduler heartbeat task statistics(avg_time=2) [2025-05-31 02:58:43.937845] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.948270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=182][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.951671] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1748678323951665, bytes: 7129513250, bw: 0.164233 MB/s, add_ts: 1005093, add_bytes: 173088 [2025-05-31 02:58:43.959163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.959838] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:43.969429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.973440] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:43.979723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:43.990075] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.000701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=176][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.012001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.022402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.036473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.045148] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=25] gc stale ls task succ [2025-05-31 02:58:44.045441] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=270] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:44.047599] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83074][T1_L0_G0][T1][YB42AC1087A6-000636654C3A382B-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525373229708, cache_obj->added_lc()=false, cache_obj->get_object_id()=53605, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:44.047743] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-05-31 02:58:44.047956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.056401] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:44.058201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.068923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.074354] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=39] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:44.082438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.085130] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2063-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:44.085172] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2063-0-0] [lt=32] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=45) [2025-05-31 02:58:44.092245] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:44.092826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=148][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.104718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.112286] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] no need to update min/max version(new_version={val:1748678323507277001, v:0}, min_version={val:1748678323507277001, v:0}, max_version={val:1748678323507277001, v:0}) [2025-05-31 02:58:44.118977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.129243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.139571] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.149925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.153422] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=11] finish to resize log external storage handler(current_ts=1748678324153416, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:58:44.153492] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=66] snapshot_traffic_info once [2025-05-31 02:58:44.160608] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.170952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.179030] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83084][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA63-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:44.181289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.194198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.204546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.217649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.220279] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137192372}], scn_array=[cnt:1, {val:1748678323908525002, v:0}], curr_size=122, accum_size=833, time_cost=15942) [2025-05-31 02:58:44.228169] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=162] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:44.228550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:44.230608] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [18113][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:44.230666] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:58:44.241076] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.245646] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:44.253219] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=181][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.263479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.274821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.281283] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=87] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:44.285148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=150][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.293492] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=31] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:44.293604] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=109] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:44.294115] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:44.294219] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109A-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:44.294245] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109A-0-0] [lt=22] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:44.295576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.307703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.313794] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678324212081000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678324313657}, version_delta=101709) [2025-05-31 02:58:44.318301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.328553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.341191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.352044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.360094] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.360163] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.360192] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.360237] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.362348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.366007] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=26] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:44.372586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.383082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.392462] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=44] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:58:44.393361] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678324393330], gts:1748678324392532000, latest_srr:[mts=1748678324393330]}) [2025-05-31 02:58:44.393432] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=69] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:44.393586] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:44.394077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.405277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.407837] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1057-0-0] [lt=63] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=33, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:44.407956] INFO [LIB] stat (utility.h:1211) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1057-0-0] [lt=117] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7161, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:44.415622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.418732] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=135] get number(ret=0, service_type_=0, range=1, base_id=1748678324418530000, start_id=1748678324418530000, end_id=1748678324418530001) [2025-05-31 02:58:44.425012] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=101] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678323908525002, v:0}, new_max_version={val:1748678323908525002, v:0}, last_min_version={val:1748678323887950000, v:0}, last_max_version={val:1748678323887950000, v:0}, total_time_us=8796, delta_us=516483) [2025-05-31 02:58:44.425083] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678323908525002, v:0}, min_version={val:1748678323908525002, v:0}, max_version={val:1748678323908525002, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:44.426510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.439601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.448335] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:44.450183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.462998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.463431] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:44.473475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.474435] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=102] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:44.485822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.495376] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:44.496300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.507009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.518085] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F3E3F8B-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1748678324515016000, start_id=1748678324515016000, end_id=1748678324515016001) [2025-05-31 02:58:44.520328] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3683) [2025-05-31 02:58:44.520856] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:44.531497] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [18113][T1_L0_G1][T1][YB42AC10871E-0006366549CC1059-0-0] [lt=133] init point thread id with(&point=0x55c0908e2400, idx_=1729, point=[thread id=18113, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=18113) [2025-05-31 02:58:44.538752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=183][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.545573] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C8-0-0] [lt=47] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:44.550851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.561289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=167][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.572141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.586472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.590670] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.590765] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.591157] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.591265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.591417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.591470] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.592136] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:44.606261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.616619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.621569] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF968-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:44.622458] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=43] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/11:303118 disk:44K/11:303118 t1m2:44K/11:303118 t1m2g0:44K/11:303118 |q0: 0 |q1: 0 [2025-05-31 02:58:44.628258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=168][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.638428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.648726] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:44.648915] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.659268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.672110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=151][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.682408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.692909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.696170] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:44.700134] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=72] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:44.703179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.713591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.719682] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678324515016000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=204648, in_cluster_service=true, cluster_version={val:1748678324392532000, v:0}, min_cluster_version={val:1748678324392532000, v:0}, max_cluster_version={val:1748678324392532000, v:0}, get_cluster_version_err=0, cluster_version_delta=327132, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195093, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=315664906, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:44.723083] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:44.723137] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=46] temporary file module use no blocks [2025-05-31 02:58:44.723151] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:44.723166] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:44.723180] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525372904240, flush_io_finished_ret_:0, flush_io_finished_round_:19892, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:44.724110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.736667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.747008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.759497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.769764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.780165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.783777] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4475-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:44.789647] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD38-0-0] [lt=11] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678324788047, keepalive_ts:-1}) [2025-05-31 02:58:44.789688] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [11506][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD38-0-0] [lt=39] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678324788047, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:44.790595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.803203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.816158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.829330] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:44.830298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.840470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.849194] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:44.851021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.864463] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.869382] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=42] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19339321344, fast_delete_log_mode=false) [2025-05-31 02:58:44.875036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.883287] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:44.883329] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=30] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:44.883354] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:44.887604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.894328] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678324894267000, start_id=1748678324894267000, end_id=1748678324894267001) [2025-05-31 02:58:44.898092] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:44.898260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.908794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.910273] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:44.910322] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52932, [delay/us]:prepare:13, schedule:266, submit:266, rt:2072, total:2618) [2025-05-31 02:58:44.910340] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:44.910398] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52932, [delay/us]: prepare:13, schedule:266, submit:266, device:2072, total:2618) [2025-05-31 02:58:44.910411] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:44.910554] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=127] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4071, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1055, total:1074) [2025-05-31 02:58:44.910583] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:44.910616] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4071, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1055, total:1074) [2025-05-31 02:58:44.910635] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:44.910649] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:44.910909] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=135] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:44.910937] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:44.910952] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:44.910967] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=162kB/s, net_out=102kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:44.910995] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] successfully alloc block(macro_id={[ver=1,mode=0,seq=399420][2nd=155]}) [2025-05-31 02:58:44.914067] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678324914064, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007704, add_bytes: 0 [2025-05-31 02:58:44.914267] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=47] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:44.919822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.930305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.936225] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=27] success to update min reserved snapshot(reserved_snapshot=1748676522139457000, duration=1800, snapshot_gc_ts=1748678322139457000) [2025-05-31 02:58:44.936385] INFO [LIB] stat (utility.h:1211) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=98, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:44.936446] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5189, avg_group_log_size=399) [2025-05-31 02:58:44.936470] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5199) [2025-05-31 02:58:44.936484] INFO [LIB] stat (utility.h:1211) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:44.939794] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3497, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:44.939907] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=111] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5189) [2025-05-31 02:58:44.940007] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=580, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:44.940175] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=381, avg_log_gen_to_submit_time=510, avg_log_submit_to_flush_time=4249, avg_log_submit_to_slide_time=5828) [2025-05-31 02:58:44.940212] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:44.940268] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5189, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:44.940303] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=16] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1748676522139457000) [2025-05-31 02:58:44.940318] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=12] update reserved snapshot finished(cost_ts=4098, reserved_snapshot=1748676522139457000) [2025-05-31 02:58:44.943767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.954093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.954180] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1748678324954176, bytes: 7129678396, bw: 0.157101 MB/s, add_ts: 1002511, add_bytes: 165146 [2025-05-31 02:58:44.965108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.965310] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=43][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:44.969142] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:44.975952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.976398] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=97] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:44.986214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:44.998629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.008270] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81431][pnio1][T0][YB42AC10870A-000636671AB9465A-0-0] [lt=12][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:275244516, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678325035173, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1380826, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678325012072, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678325008264, enqueue_timestamp_:0, request_arrival_time_:1748678325008264, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:45.008350] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81431][pnio1][T0][YB42AC10870A-000636671AB9465A-0-0] [lt=62][errcode=-5150] deliver rpc request fail(&req=0x2b0959604098, ret=-5150) [2025-05-31 02:58:45.009185] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.019423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.025912] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83091][T1_L0_G0][T1][YB42AC10870A-000636671538EF37-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1748678325022051000, start_id=1748678325022051000, end_id=1748678325022051001) [2025-05-31 02:58:45.026006] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678324660075000, v:0}, base_version={val:1748678324621794000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678324823173000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678325025945, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678324660075000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678324932289, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678324689286001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678324963876, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678320277136000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678325036710, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678320002471000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678324954822, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678320186899000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678324966759, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:58:45.026688] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=53] ====== check clog disk timer task ====== [2025-05-31 02:58:45.026713] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:45.030989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.041458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=152][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.045349] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=17] gc stale ls task succ [2025-05-31 02:58:45.048489] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=21] current monitor number(seq_=-1) [2025-05-31 02:58:45.049416] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:45.053566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.056633] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:45.056716] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A382D-0-0] [lt=39][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525374238885, cache_obj->added_lc()=false, cache_obj->get_object_id()=53606, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:45.064214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.077131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.080264] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:45.090376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.092033] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81740][T1_TimerWK0_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:45.098780] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=59] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:45.100293] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=629140, clean_num=31457) [2025-05-31 02:58:45.100960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.111155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.121532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.131937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.142277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.152648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.162882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.173095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.183493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=200][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.193973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.202446] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83104][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA64-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:45.208764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.223666] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA6B-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525374405834, cache_obj->added_lc()=false, cache_obj->get_object_id()=53607, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:45.224430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.239507] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.239557] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:45.241576] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [17084][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:58:45.253162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.253794] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=204464, replace_num=15728) [2025-05-31 02:58:45.253875] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=79] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:45.263576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.274180] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.285011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.297930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.301178] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:45.306929] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=32] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:45.308949] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=68] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:45.309140] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=51] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:45.309180] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=37] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678322139457000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:58:45.310321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.321310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.331941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=277][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.342184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.352461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.358971] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.359025] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.359343] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.359388] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.363110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.366569] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [81064][TimerWK2_RSInspectTask][T0][Y0-0000000000000000-0-0] [lt=37] add async task(this=task_type:N9oceanbase10rootserver21ObPurgeRecyclebinTaskE, timeout_check:False) [2025-05-31 02:58:45.373655] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.383977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.395473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.395633] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=37] ts response task statistics(alloc_count=485954, free_count=242976) [2025-05-31 02:58:45.395666] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=644, get_gts_with_stc_cnt=62, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=60, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:45.395684] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:45.395726] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678324894389], gts:1748678324928466000, latest_srr:[mts=1748678325395698]}) [2025-05-31 02:58:45.406364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.413766] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:45.413804] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5453, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:45.417396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.428184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.436402] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678325028286001, v:0}, new_max_version={val:1748678325028286001, v:0}, last_min_version={val:1748678324894267000, v:0}, last_max_version={val:1748678324894267000, v:0}, total_time_us=9739, delta_us=408112) [2025-05-31 02:58:45.436467] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678325028286001, v:0}, min_version={val:1748678325028286001, v:0}, max_version={val:1748678325028286001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:45.438487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.452094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.454176] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC108714-000636671E6D5123-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525374636346, cache_obj->added_lc()=false, cache_obj->get_object_id()=53608, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:45.456755] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:45.463943] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.470455] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=50] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:45.474768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.477477] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:45.485501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.496244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.501707] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:45.506461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.516699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.520597] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=32] ====== checkpoint timer task ====== [2025-05-31 02:58:45.521540] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:45.521593] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:45.522076] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:45.522189] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=107] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:45.522253] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=55] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:45.522270] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678323711104001, v:0}) [2025-05-31 02:58:45.522279] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:45.522287] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:45.522303] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:45.522324] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:45.522338] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:45.522360] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:45.522382] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:45.522393] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:45.527580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.538025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.548294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.558573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.569159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.580360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.584837] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A68-0-0] [lt=51] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:45.590356] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A7-00063665469F7B36-0-0] [lt=47] get number(ret=0, service_type_=0, range=1, base_id=1748678325589297000, start_id=1748678325589297000, end_id=1748678325589297001) [2025-05-31 02:58:45.591105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.592430] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=57] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592506] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=87] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592537] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=106] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=96] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.592744] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=83] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:45.601433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.612116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.614301] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1067-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5736) [2025-05-31 02:58:45.622278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.630302] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF977-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:45.631278] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:64K/14:303387 disk:64K/14:303387 t1m2:64K/14:303387 t1m2g0:64K/14:303387 |q0: 0 |q1: 0 [2025-05-31 02:58:45.635780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.636519] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x55c090a74fc0, last_check_time:1748678325636515, last_idle_time:78702037201, last_thread_count:4}, *pool={name:TimerWK, this:0x55c090a74fc0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:78702037201, tenant_id:500}, qsize=-4, current_idle_time=78702037201) [2025-05-31 02:58:45.636557] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=38] [thread pool stat](pool_stat={pool:0x2b08f13edcf0, last_check_time:1748678325636556, last_idle_time:42470507053, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13edcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:1, threads_idle_time:42470507053, tenant_id:500}, qsize=0, current_idle_time=42470507053) [2025-05-31 02:58:45.636570] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x2b08f13eeaf0, last_check_time:1748678325636569, last_idle_time:19941211740, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13eeaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:19941211740, tenant_id:500}, qsize=-1, current_idle_time=19941211740) [2025-05-31 02:58:45.636581] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x2b08efdfee70, last_check_time:1748678325636580, last_idle_time:77211311675, last_thread_count:4}, *pool={name:TimerWK, this:0x2b08efdfee70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:77211311675, tenant_id:1}, qsize=-4, current_idle_time=77211311675) [2025-05-31 02:58:45.636956] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=373] [thread pool stat](pool_stat={pool:0x2b08f13ef1f0, last_check_time:1748678325636955, last_idle_time:2768522066, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2768522066, tenant_id:500}, qsize=-1, current_idle_time=2768522066) [2025-05-31 02:58:45.636971] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] [thread pool stat](pool_stat={pool:0x2b08f13ef8f0, last_check_time:1748678325636971, last_idle_time:2768100591, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef8f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2768100591, tenant_id:500}, qsize=-1, current_idle_time=2768100591) [2025-05-31 02:58:45.646142] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.658256] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:45.658574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.670632] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.680949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.692620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.703063] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:45.703290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.711463] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=59] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:45.711498] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=32] try gc retain ctx [2025-05-31 02:58:45.711507] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-31 02:58:45.711607] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678325630415000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678315524251001, v:0}, keep_alive_scn_:{val:1748678320473655001, v:0}, update_ts_:1748678325710632}) [2025-05-31 02:58:45.712370] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:45.714114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.723643] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:45.723696] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] temporary file module use no blocks [2025-05-31 02:58:45.723707] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:45.723723] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:45.723739] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525373905394, flush_io_finished_ret_:0, flush_io_finished_round_:19893, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:45.725549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.736379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.749009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.760014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.770701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.782857] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.786989] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4476-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:45.793273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.803490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.813706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.824239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=329][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.834951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.847294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.859712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.859966] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:45.872580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.877613] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=40] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:58:45.882922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.894501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.897125] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:45.897263] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=134] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678325396014], gts:1748678325430826000, latest_srr:[mts=1748678325897085]}) [2025-05-31 02:58:45.904752] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:45.904912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.914469] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=44] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:45.915802] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=139] PNIO [ratelimit] time: 1748678325915798, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001734, add_bytes: 0 [2025-05-31 02:58:45.915979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.926568] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.934003] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:45.934070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56013, [delay/us]:prepare:29, schedule:408, submit:372, rt:1999, total:2810) [2025-05-31 02:58:45.934090] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:45.934150] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56013, [delay/us]: prepare:29, schedule:408, submit:372, device:1999, total:2810) [2025-05-31 02:58:45.934164] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:45.934191] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4001, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1063, total:1078) [2025-05-31 02:58:45.934215] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:45.934244] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4001, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1063, total:1078) [2025-05-31 02:58:45.934262] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:45.934276] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:45.934285] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:45.934294] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:45.934302] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:45.934311] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=143kB/s, net_out=97kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:45.934332] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399421][2nd=2591]}) [2025-05-31 02:58:45.936932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.947376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.958171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.958555] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1748678325958551, bytes: 7129828628, bw: 0.142648 MB/s, add_ts: 1004375, add_bytes: 150232 [2025-05-31 02:58:45.968582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.973184] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:45.981052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:45.981200] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:45.991463] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.002257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.013372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.014167] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=86] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=92, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:46.014207] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=4336, avg_group_log_size=333) [2025-05-31 02:58:46.014228] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4326) [2025-05-31 02:58:46.014243] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:46.022743] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3295, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:46.022781] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4336) [2025-05-31 02:58:46.022806] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1076, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:46.023105] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=122, avg_log_submit_to_flush_time=4713, avg_log_submit_to_slide_time=6011) [2025-05-31 02:58:46.023143] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=48, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:46.023227] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4336, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:46.024168] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.034675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.037135] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:46.045630] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=25] gc stale ls task succ [2025-05-31 02:58:46.047509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.048650] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2025-05-31 02:58:46.057741] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:46.058405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.065954] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A382F-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525375248121, cache_obj->added_lc()=false, cache_obj->get_object_id()=53609, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.066089] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:46.069242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.080787] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:46.081415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.090153] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2064-0-0] [lt=595] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:46.090195] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2064-0-0] [lt=32] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=43) [2025-05-31 02:58:46.096817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.105618] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:46.108778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.120676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.131359] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AE4-0-0] [lt=63] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:46.131646] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.141982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.152283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.153995] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B27-0-0] [lt=7] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:46.160696] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=89] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:46.160792] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=32] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:46.160813] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=953) [2025-05-31 02:58:46.164013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.174315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.184644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.195245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=287][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.206721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.217154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.227455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.238723] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA65-0-0] [lt=91][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:46.239135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.243125] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=46] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:46.249422] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.249475] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:46.249535] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [17667][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=63] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:58:46.260014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.272232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.275006] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:46.277760] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC10870A-000636671448C590-0-0] [lt=71] get number(ret=0, service_type_=0, range=1, base_id=1748678326276770000, start_id=1748678326276770000, end_id=1748678326276770001) [2025-05-31 02:58:46.286019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.294587] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=24] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:46.294611] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:46.294715] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3C-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:46.294739] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3C-0-0] [lt=19] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:46.304370] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:46.306333] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:46.306708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.317627] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=51] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:46.319800] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=83] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:46.323346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.324204] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=21] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:46.324233] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=28] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:46.326713] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=9] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:46.326793] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=29] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:46.326810] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=14] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:46.326837] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:46.327434] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=592] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:46.327502] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=58] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:46.327514] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:46.327564] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=46] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:46.327612] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=47] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:46.329303] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:46.329326] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=21] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:46.329336] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:46.329373] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=6] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:46.329386] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:46.329395] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:46.329408] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:46.329424] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD8-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:46.334010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.344233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.354532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.358931] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.358994] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.359024] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.359050] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.364863] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.376584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.386999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.401110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.403481] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3831-0-0] [lt=137][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525375585640, cache_obj->added_lc()=false, cache_obj->get_object_id()=53610, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.408486] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3833-0-0] [lt=37][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:46.409414] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3835-0-0] [lt=39][errcode=-4201] hashtable not init [2025-05-31 02:58:46.409531] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=15] get number(ret=0, service_type_=0, range=1, base_id=1748678326409267000, start_id=1748678326409267000, end_id=1748678326409267001) [2025-05-31 02:58:46.411831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.424191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.437066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.443352] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=10, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:46.443387] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5570, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:46.443636] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678326025930001, v:0}, new_max_version={val:1748678326025930001, v:0}, last_min_version={val:1748678325896596000, v:0}, last_max_version={val:1748678325896596000, v:0}, total_time_us=8701, delta_us=417702) [2025-05-31 02:58:46.443722] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=81] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678326025930001, v:0}, min_version={val:1748678326025930001, v:0}, max_version={val:1748678326025930001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:46.447620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.455652] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC1087A7-0006366541CDB88C-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1748678326454828000, start_id=1748678326454828000, end_id=1748678326454828001) [2025-05-31 02:58:46.457878] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.468133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.474148] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:46.475299] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:46.478428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.482187] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:46.488725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.499994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.507826] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:46.510230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.520489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.533428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.544035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.555955] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.557804] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20C9-0-0] [lt=27] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:46.566747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.582968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.590072] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.590125] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.590154] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.591744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.591904] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.592133] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.592323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:46.593384] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.594536] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:46.594619] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.594669] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.594758] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.602654] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12] ls blacklist refresh finish(cost_time=33722) [2025-05-31 02:58:46.603801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.616603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.627199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.627977] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=38] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:251830 disk:44K/10:251830 t1m2:44K/10:251830 t1m2g0:44K/10:251830 |q0: 0 |q1: 0 [2025-05-31 02:58:46.632260] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4110) [2025-05-31 02:58:46.637502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.640417] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF986-0-0] [lt=20] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:46.648079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.659374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.664241] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=32] ====== traversal_flush timer task ====== [2025-05-31 02:58:46.665313] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60BB-0-0] [lt=41] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678326637771000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:46.665506] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60BB-0-0] [lt=182] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:46.670110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.673274] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3D4-0-0] [lt=75][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525375855443, cache_obj->added_lc()=false, cache_obj->get_object_id()=53611, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:46.678019] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:46.681285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.684132] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:46.686309] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81743][T1_TimerWK2_ReqMemEvict][T1][YB42AC1087A6-00063665430A5EBD-0-0] [lt=67] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=50) [2025-05-31 02:58:46.686930] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=112] get number(ret=0, service_type_=0, range=1, base_id=1748678326686010000, start_id=1748678326686010000, end_id=1748678326686010001) [2025-05-31 02:58:46.692029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.702277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.708809] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=120] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:46.712746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.723050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.724241] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:46.724293] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=43] temporary file module use no blocks [2025-05-31 02:58:46.724344] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=48] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:46.724363] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:46.724378] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525374905954, flush_io_finished_ret_:0, flush_io_finished_round_:19894, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:46.733408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.743682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.755057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.755681] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=21] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:46.755733] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=32] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:46.755749] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:46.755759] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:46.758833] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:46.760312] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=1473] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:46.765621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.776101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.787198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.790219] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4477-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:46.799068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.800439] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=10] ====== tenant freeze timer task ====== [2025-05-31 02:58:46.800496] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=49] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:46.800565] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:46.812575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.825391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.838533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.850187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=149][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.860468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.866772] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=39] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:46.872589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.879242] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:46.885647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.895800] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.899739] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:46.906248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=206][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.909285] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:46.917098] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=43] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:46.917235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.922192] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=98] PNIO [ratelimit] time: 1748678326922188, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006390, add_bytes: 0 [2025-05-31 02:58:46.927389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.937672] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.945909] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:46.945981] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48573, [delay/us]:prepare:13, schedule:378, submit:45, rt:3354, total:3791) [2025-05-31 02:58:46.946002] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:46.946061] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48573, [delay/us]: prepare:13, schedule:378, submit:45, device:3354, total:3791) [2025-05-31 02:58:46.946075] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:46.946219] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=111] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4047, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3651, total:3728) [2025-05-31 02:58:46.946253] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:46.946404] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=142] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4046, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3651, total:3728) [2025-05-31 02:58:46.946430] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:46.946448] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:46.946457] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:46.946466] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:46.946474] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:46.946483] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=265kB/s, net_out=188kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:46.946503] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399422][2nd=3027]}) [2025-05-31 02:58:46.950772] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.959023] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1748678326959018, bytes: 7130098698, bw: 0.257439 MB/s, add_ts: 1000467, add_bytes: 270070 [2025-05-31 02:58:46.961372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.972726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.974474] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-31 02:58:46.983820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:46.985825] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:46.994112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.004805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.006552] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [16802][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9B-0-0] [lt=38] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678327038923, keepalive_ts:-1}) [2025-05-31 02:58:47.006693] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [16802][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9B-0-0] [lt=138] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678327038923, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:47.015595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=190][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.022009] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81060][TimerWK0_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=26] runTimer to refresh locality_info(ret=0) [2025-05-31 02:58:47.026309] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:58:47.026410] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=85] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:58:47.026424] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=12] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:58:47.026676] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=232] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:58:47.026706] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=30] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:58:47.026789] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=65] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:58:47.026810] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=20] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:58:47.026882] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=64] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:58:47.028024] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.028330] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=54] ====== check clog disk timer task ====== [2025-05-31 02:58:47.028358] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:47.033471] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:47.033534] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4546, avg_group_log_size=413) [2025-05-31 02:58:47.033558] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4546) [2025-05-31 02:58:47.033573] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:47.035754] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3570, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:47.035784] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4546) [2025-05-31 02:58:47.035807] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=385, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:47.035973] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:47.036303] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC1077-0-0] [lt=133] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=4011, avg_log_submit_to_slide_time=4923) [2025-05-31 02:58:47.037108] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4546, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:47.041395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:47.044360] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [19015][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:47.047543] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:58:47.049900] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=57] gc stale ls task succ [2025-05-31 02:58:47.052691] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:47.057740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.057967] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:47.067928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.079044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.080367] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:47.080968] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:47.083261] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3837-0-0] [lt=17][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525376265431, cache_obj->added_lc()=false, cache_obj->get_object_id()=53612, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:47.090573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.106154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.109275] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33BB-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525376291444, cache_obj->added_lc()=false, cache_obj->get_object_id()=53613, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:47.109966] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:47.111363] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=660597, clean_num=31457) [2025-05-31 02:58:47.117624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.128127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.139226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.149701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.164510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.177187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.189475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.200233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=142][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.210517] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.222029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.232564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.246581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.256294] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=83] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:47.257941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.270475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.281951] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA66-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:47.282373] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:47.282494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.293257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.303573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.312039] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:47.316668] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.327777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.339701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.352241] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.359284] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.359335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.359363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.359389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.365344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.366511] INFO pn_ratelimit (group.c:653) [81065][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=26] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:47.375918] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.386403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.398747] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83090][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3839-0-0] [lt=33][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525376580914, cache_obj->added_lc()=false, cache_obj->get_object_id()=53614, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:47.399442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.400605] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678327400507], gts:1748678327399957000, latest_srr:[mts=1748678327400507]}) [2025-05-31 02:58:47.409838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.420918] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.431140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.441414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.448017] INFO [LIB] stat (utility.h:1211) [10191][T1_L0_G1][T1][YB42AC10871E-0006366549CC107B-0-0] [lt=127] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:47.448051] INFO [LIB] stat (utility.h:1211) [10191][T1_L0_G1][T1][YB42AC10871E-0006366549CC107B-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6655, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:47.451961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.453316] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678327040846000, v:0}, new_max_version={val:1748678327040846000, v:0}, last_min_version={val:1748678326899424000, v:0}, last_max_version={val:1748678326899424000, v:0}, total_time_us=4962, delta_us=412466) [2025-05-31 02:58:47.453437] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=115] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678327040846000, v:0}, min_version={val:1748678327040846000, v:0}, max_version={val:1748678327040846000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:47.462664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.473026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.476177] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:47.483268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.485802] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=220192, replace_num=15728) [2025-05-31 02:58:47.485834] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:47.486343] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:47.495157] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.505362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.513400] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:47.515665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.528281] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.538777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.548579] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89] get_max_applied_scn(scn={val:1748678327450354000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137207743}, palf_committed_end_scn_:{val:1748678327450354000, v:0}, last_check_scn_:{val:1748678327450354000, v:0}, max_applied_cb_scn_:{val:1748678327450354000, v:0}}) [2025-05-31 02:58:47.548631] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137207743}, readable_log_tail:{lsn:2137207743}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10144833, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:47.549360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.561466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.571740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.577310] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81060][TimerWK0_ServerTracerTim][T0][YB42AC1087A6-0006366547BA593F-0-0] [lt=18] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:47.581969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.589934] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.590912] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=73] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.590952] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=105] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.590980] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.591008] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.591034] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.592220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.592358] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:47.602430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.603772] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5979) [81743][T1_TimerWK2_SSTableGC][T1][YB42AC1087A6-00063665430A5EC0-0-0] [lt=24] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1748671375638251000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1748676522139457000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1748676522139457000) [2025-05-31 02:58:47.613052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.623292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.634310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.644754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.648547] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [11506][T1_L0_G1][T1][YB42AC10871E-0006366549CC107D-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4103) [2025-05-31 02:58:47.650595] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF995-0-0] [lt=186] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.676903343200683594e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.677807060445658408e+03) [2025-05-31 02:58:47.651562] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=76] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:56K/13:536930 disk:56K/13:536930 t1m2:56K/13:536930 t1m2g0:56K/13:536930 |q0: 0 |q1: 0 [2025-05-31 02:58:47.655144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.665503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.678543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.689061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=190][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.695063] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=144] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:47.699628] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:113) [81743][T1_TimerWK2_SSTableGC][T1][YB42AC1087A6-00063665430A5EC0-0-0] [lt=58] SSTableGCTask(cost_ts=96776) [2025-05-31 02:58:47.700533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.710303] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27] LogLoopThread round_cost_time(us)(round_cost_time=22) [2025-05-31 02:58:47.710766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.714862] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:47.721190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.724543] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:47.724594] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=42] temporary file module use no blocks [2025-05-31 02:58:47.724606] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:47.724621] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:47.724636] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525375906592, flush_io_finished_ret_:0, flush_io_finished_round_:19895, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:47.731492] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.742032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.752290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.762561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.772806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.784781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.795323] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4478-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:47.796430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.807137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.817620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.828616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.840341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.852762] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.865654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.875917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.887513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.898225] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=106] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:47.898389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.903164] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:47.909141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.917846] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:47.918111] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=46] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:47.919459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.928657] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=94] PNIO [ratelimit] time: 1748678327928652, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006464, add_bytes: 0 [2025-05-31 02:58:47.929938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.940201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.952488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.952613] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=29] [APPLY STAT CB APPEND COST TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=152, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:47.952644] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=30] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=6644, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:47.952653] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:47.952660] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=-1162, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:47.952667] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3836-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=52, stat_interval=5000000, avg (count/cost)=5634, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:47.961054] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:47.961117] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48419, [delay/us]:prepare:23, schedule:355, submit:54, rt:2128, total:2562) [2025-05-31 02:58:47.961136] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:47.961216] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=63] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48418, [delay/us]: prepare:23, schedule:355, submit:54, device:2128, total:2562) [2025-05-31 02:58:47.961231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:47.961258] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4035, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:470, total:486) [2025-05-31 02:58:47.961285] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:47.961316] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4035, [delay/us]: prepare:0, schedule:0, submit:600000000, device:470, total:486) [2025-05-31 02:58:47.961334] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:47.961348] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:47.961357] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:47.961366] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:47.961374] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:47.961428] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=252kB/s, net_out=224kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:47.961455] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] successfully alloc block(macro_id={[ver=1,mode=0,seq=399423][2nd=2400]}) [2025-05-31 02:58:47.963290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.967767] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1748678327967764, bytes: 7130355990, bw: 0.243245 MB/s, add_ts: 1008746, add_bytes: 257292 [2025-05-31 02:58:47.973575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.977679] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:47.983755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:47.990435] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:47.993940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.005581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.011109] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:48.015228] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:58:48.015417] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=31] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:58:48.016673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.028559] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.041038] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.049623] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:48.049666] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4103, avg_group_log_size=373) [2025-05-31 02:58:48.049687] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4103) [2025-05-31 02:58:48.049702] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:48.049783] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-05-31 02:58:48.050128] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=27] gc stale ls task succ [2025-05-31 02:58:48.051139] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2552, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:48.051167] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4103) [2025-05-31 02:58:48.051188] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=375, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:48.051223] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:48.053730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.053784] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:48.055352] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [11506][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=32] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:58:48.058797] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:48.059034] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [16802][T1_L0_G1][T1][YB42AC10871E-0006366549CC1082-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=36, avg_log_gen_to_submit_time=124, avg_log_submit_to_flush_time=2974, avg_log_submit_to_slide_time=6496) [2025-05-31 02:58:48.059267] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=103] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4103, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:48.065243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.076820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.082729] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:48.088161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.093822] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A383B-0-0] [lt=53][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525377275991, cache_obj->added_lc()=false, cache_obj->get_object_id()=53615, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:48.098594] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:48.099979] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2065-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:48.100037] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2065-0-0] [lt=47] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=644) [2025-05-31 02:58:48.100584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.108370] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=35] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:48.113501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.118568] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=113] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:48.126060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.136314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.148463] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.159151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.169426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.179729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.190264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=221][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.200920] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.211478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.224805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.235191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=146][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.246010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.251150] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=43] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:48.256371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.263727] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:48.268269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.278506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.291349] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.291586] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA545E-0-0] [lt=49] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678323580352, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:48.291636] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA545E-0-0] [lt=49] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678323582488, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:48.291661] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA545E-0-0] [lt=20] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678323594290, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678323594298, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:48.301756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.303059] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=36] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:48.303098] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=38] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:48.303231] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=134] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:48.303964] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3D-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:48.303989] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3D-0-0] [lt=22] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:48.307259] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA67-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:48.313953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.318446] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:48.319093] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:48.320659] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:48.320794] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=44] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:48.324391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.335462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.348233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.358601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.363990] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.364038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.364065] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.364112] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.368918] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.379178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.389310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.399533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.404018] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=49] ts response task statistics(alloc_count=486026, free_count=243012) [2025-05-31 02:58:48.404114] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678327903215], gts:1748678327937480000, latest_srr:[mts=1748678328404072]}) [2025-05-31 02:58:48.412665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.423558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.433836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.445378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=256][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.458137] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1748678328457849000, start_id=1748678328457849000, end_id=1748678328457849001) [2025-05-31 02:58:48.458624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.466997] INFO [LIB] stat (utility.h:1211) [16802][T1_L0_G1][T1][YB42AC10871E-0006366549CC1086-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=28, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:48.467210] INFO [LIB] stat (utility.h:1211) [16802][T1_L0_G1][T1][YB42AC10871E-0006366549CC1086-0-0] [lt=211] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5950, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:48.468912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.469828] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [19015][T1_L0_G1][T1][YB42AC10871E-0006366549CC1087-0-0] [lt=25] init point thread id with(&point=0x55c0908f0580, idx_=2631, point=[thread id=19015, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=19015) [2025-05-31 02:58:48.470651] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678328053515001, v:0}, new_max_version={val:1748678328053515001, v:0}, last_min_version={val:1748678327940510000, v:0}, last_max_version={val:1748678327940510000, v:0}, total_time_us=14944, delta_us=417132) [2025-05-31 02:58:48.470696] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678328053515001, v:0}, min_version={val:1748678328053515001, v:0}, max_version={val:1748678328053515001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:48.479975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.481056] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=150] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:48.491597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.491764] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:48.505691] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:48.505742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.516122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.521352] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:48.526566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.537370] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.547633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.558452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.566762] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CA-0-0] [lt=188] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:48.570402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.582206] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A29-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525377764376, cache_obj->added_lc()=false, cache_obj->get_object_id()=53616, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:48.585739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.593802] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.593919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.593947] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.593974] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.594000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.594024] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.595550] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:48.597735] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83088][T1_L0_G0][T1][YB42AC108714-000636671E2D3A69-0-0] [lt=49] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:48.598148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.608984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.619790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.632378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=146][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.642540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.652899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.658325] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF99E-0-0] [lt=15] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.676903343200683594e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.677807060445658408e+03) [2025-05-31 02:58:48.658785] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:569588 disk:44K/10:569588 t1m2:44K/10:569588 t1m2g0:44K/10:569588 |q0: 0 |q1: 0 [2025-05-31 02:58:48.661253] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [16802][T1_L0_G1][T1][YB42AC1087A7-00063665517F54C2-0-0] [lt=81] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4103) [2025-05-31 02:58:48.665796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.669564] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=22] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:48.669617] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=38] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=986) [2025-05-31 02:58:48.676070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.686353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.696760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.706194] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=74] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:48.709291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.719598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.723001] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:48.724912] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=65] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:48.724966] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-05-31 02:58:48.724996] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:48.725011] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:48.725036] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525376906887, flush_io_finished_ret_:0, flush_io_finished_round_:19896, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:48.729869] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.740045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.750551] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.760807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.771127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=157][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.781320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.791456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.799293] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4479-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:48.802404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.811637] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [16802][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD39-0-0] [lt=30] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678328810449, keepalive_ts:-1}) [2025-05-31 02:58:48.811692] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [16802][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD39-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678328810449, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:48.813023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.817582] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] ====== tenant freeze timer task ====== [2025-05-31 02:58:48.817617] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=25] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:48.817642] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:48.818224] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:48.818498] WDIAG read_multi_row (ob_table_access_helper.h:238) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:48.820573] INFO delete_row (ob_table_access_helper.h:339) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=40] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678328818539) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678328818539) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:58:48.824968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.835511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.848205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.859598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.871378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:48.874995] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [19591][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:48.875053] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:58:48.877004] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:115) [81743][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=15] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1, avail_memory=53687091, chunk_count_=0, avail_sort_memory_=53687091, avail_memory_=53687091) [2025-05-31 02:58:48.887352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.897581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.897712] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81740][T1_TimerWK0_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=34] clear dtl interm result cost(us)(clear_cost=4278, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:58:48.906600] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1748678328905930000, start_id=1748678328905930000, end_id=1748678328905930001) [2025-05-31 02:58:48.906674] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=56] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:48.907130] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678328404415], gts:1748678328438533000, latest_srr:[mts=1748678328907083]}) [2025-05-31 02:58:48.907489] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:48.907910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.918320] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:48.919575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.925984] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:48.930355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.930398] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:58:48.932544] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=34] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:58:48.936282] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1748678328936264, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-05-31 02:58:48.940576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.950961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.961277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.967259] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:48.967322] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44778, [delay/us]:prepare:11, schedule:847, submit:64, rt:2040, total:2963) [2025-05-31 02:58:48.967342] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:48.967402] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44778, [delay/us]: prepare:11, schedule:847, submit:64, device:2040, total:2963) [2025-05-31 02:58:48.967416] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:48.967442] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4070, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:367, total:382) [2025-05-31 02:58:48.967468] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:48.967496] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4070, [delay/us]: prepare:0, schedule:0, submit:600000000, device:367, total:382) [2025-05-31 02:58:48.967514] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:48.967529] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:48.967538] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:48.967546] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:48.967554] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:48.967608] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS SERVER] net_in=164kB/s, net_out=98kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:48.967632] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] successfully alloc block(macro_id={[ver=1,mode=0,seq=399424][2nd=2823]}) [2025-05-31 02:58:48.971546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.972965] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1748678328972962, bytes: 7130525089, bw: 0.160431 MB/s, add_ts: 1005198, add_bytes: 169099 [2025-05-31 02:58:48.981520] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=83] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:48.982091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:48.992627] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:48.992789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.005714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.016202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.017663] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=27] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:49.026382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.030651] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-05-31 02:58:49.030689] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:49.036802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.047590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.050402] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=18] current monitor number(seq_=-1) [2025-05-31 02:58:49.050508] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:58:49.057080] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=119, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:49.057139] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3555, avg_group_log_size=355) [2025-05-31 02:58:49.057209] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3555) [2025-05-31 02:58:49.057231] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:49.058568] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3225, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:49.058595] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3555) [2025-05-31 02:58:49.058618] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=786, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:49.058654] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=91, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:49.061118] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [19591][T1_L0_G1][T1][YB42AC1087A7-00063665517F54C6-0-0] [lt=130] init point thread id with(&point=0x55c0908f9580, idx_=3207, point=[thread id=19591, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=19591) [2025-05-31 02:58:49.061299] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [19591][T1_L0_G1][T1][YB42AC1087A7-00063665517F54C6-0-0] [lt=97] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=1178, avg_log_gen_to_submit_time=1320, avg_log_submit_to_flush_time=4046, avg_log_submit_to_slide_time=5695) [2025-05-31 02:58:49.061622] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3555, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:49.064618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.064788] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:49.074954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.082898] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:49.086284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.096986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.106293] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A383D-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525378288461, cache_obj->added_lc()=false, cache_obj->get_object_id()=53617, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:49.107652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.110062] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=222] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:49.120830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.126973] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:49.128549] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=692054, clean_num=31457) [2025-05-31 02:58:49.131239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.141581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.153231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.165047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.171054] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AE6-0-0] [lt=56] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:49.177786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.188103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.199676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.210374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.220721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.230938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.241178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.249682] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0D52-0-0] [lt=31] handle gts request statistics(total_rt=1243, total_cnt=189, avg_rt=6.542105263157894868e+00, avg_cnt=1.889999999999999858e+01) [2025-05-31 02:58:49.251435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.261707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.271917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.283069] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=98] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:49.283238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.294229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.304670] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.313280] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:49.316631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.326437] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA68-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:49.326956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.329121] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:49.337867] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.348558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.359048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.361491] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.361555] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.361588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.361615] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.365422] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678329264455000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678329365367}, version_delta=100963) [2025-05-31 02:58:49.369981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.380240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.392174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.404101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.405972] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=63] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:58:49.407705] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=2, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:49.415450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.425676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.436151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.447073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.458533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.471109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.473644] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:49.473763] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=116] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5880, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:49.474425] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678329066219001, v:0}, new_max_version={val:1748678329066219001, v:0}, last_min_version={val:1748678328905930000, v:0}, last_max_version={val:1748678328905930000, v:0}, total_time_us=6463, delta_us=408202) [2025-05-31 02:58:49.474465] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678329066219001, v:0}, min_version={val:1748678329066219001, v:0}, max_version={val:1748678329066219001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:49.474490] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:49.481504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.484594] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=57] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:49.491800] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.494969] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=109] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:49.502335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.514184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.514347] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:49.525342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.529659] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:49.535633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.546211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.546705] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F7E3EB5-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678329546401000, start_id=1748678329546401000, end_id=1748678329546401001) [2025-05-31 02:58:49.556683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.567203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.577480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.587718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.590966] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591024] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591050] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591093] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591148] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591199] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.591249] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:49.595004] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=226][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:49.595060] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:49.595105] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:49.595128] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:49.598017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.608298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.608905] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] start to print blacklist info [2025-05-31 02:58:49.609356] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=120] ls blacklist refresh finish(cost_time=39748) [2025-05-31 02:58:49.619146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.629382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.643546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.654401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.662777] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [19591][T1_L0_G1][T1][YB42AC1087A7-00063665517F54CD-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4225) [2025-05-31 02:58:49.664659] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.671309] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9A9-0-0] [lt=16] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=285, Holding Memory(MB)=134, Tenant Remain Memory(MB)=251, Usable Remain Memory(MB)=150, Last Update Limit Timestamp=1748678329670761, Is Updated=true) [2025-05-31 02:58:49.671368] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9A9-0-0] [lt=56] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.700903892517089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633381041092147825e+03) [2025-05-31 02:58:49.673915] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC108714-000636671E7D5D66-0-0] [lt=48] get number(ret=0, service_type_=0, range=1, base_id=1748678329672156000, start_id=1748678329672156000, end_id=1748678329672156001) [2025-05-31 02:58:49.675500] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:743108 disk:52K/12:743108 t1m2:52K/12:743108 t1m2g0:52K/12:743108 |q0: 0 |q1: 0 [2025-05-31 02:58:49.675842] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=236][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.686137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.696902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.708625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=131][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.715548] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=235920, replace_num=15728) [2025-05-31 02:58:49.715605] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:49.717015] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678319704301, end_time=1748678329717011, dag_status={}) [2025-05-31 02:58:49.717065] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=46] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717082] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717091] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717120] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717127] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717134] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:49.717143] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:58:49.717152] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:58:49.719269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.726361] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:49.726413] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-05-31 02:58:49.726426] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:49.726441] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:49.726456] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525377907251, flush_io_finished_ret_:0, flush_io_finished_round_:19897, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:49.729536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.731015] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:49.741091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.751763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.764960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.769001] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678329546401000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=222586, in_cluster_service=true, cluster_version={val:1748678329404619000, v:0}, min_cluster_version={val:1748678329404619000, v:0}, max_cluster_version={val:1748678329404619000, v:0}, get_cluster_version_err=0, cluster_version_delta=364368, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195143, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=320714229, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:49.775255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.785959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.796229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.802420] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447A-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:49.806505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.816891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.829282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.833246] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=170] BlockGCTimerTask success(ret=0, cost_time_us=17, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:58:49.839921] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.852421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.863531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.872037] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=79] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19338551296, fast_delete_log_mode=false) [2025-05-31 02:58:49.873824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.874044] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:49.874176] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [19015][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=59] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:58:49.885208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.895458] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:74, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183659088966, recv_hp_rpc_cnt:74878, recv_np_rpc_cnt:877075, recv_lp_rpc_cnt:0, recv_mysql_cnt:46975, recv_task_cnt:2470, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1230962 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9870, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:543577, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1748678329874039}{group_id:10, queue_size:0, recv_req_cnt:103853, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18345, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678322476321}{group_id:2, queue_size:0, recv_req_cnt:150787, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101009, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:107824, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272276, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=676 pcode=0x717:cnt=477 pcode=0x1502:cnt=270 pcode=0x710:cnt=174 pcode=0x523:cnt=97, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:58:49.897327] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1864] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82634701, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:58:49.898392] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1062] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:85133519, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:58:49.899063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=661][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.908809] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:49.910329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.918461] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:49.918745] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=58] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:49.922625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.931452] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:49.933295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.936331] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=201] PNIO [ratelimit] time: 1748678329936326, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000062, add_bytes: 0 [2025-05-31 02:58:49.945546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.956429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.968686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.970269] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=25] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:58:49.972993] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1748678329972988, bytes: 7130699421, bw: 0.166252 MB/s, add_ts: 1000026, add_bytes: 174332 [2025-05-31 02:58:49.975948] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:49.976031] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56848, [delay/us]:prepare:16, schedule:1018, submit:166, rt:1809, total:3010) [2025-05-31 02:58:49.976114] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:49.976255] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56841, [delay/us]: prepare:16, schedule:1018, submit:166, device:1809, total:3010) [2025-05-31 02:58:49.976273] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:49.976300] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4060, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:380, total:395) [2025-05-31 02:58:49.976366] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:49.976405] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4059, [delay/us]: prepare:0, schedule:0, submit:600000000, device:380, total:395) [2025-05-31 02:58:49.976424] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:49.976439] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:49.976447] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:49.976456] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:49.976464] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:49.976472] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=170kB/s, net_out=144kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:49.976493] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399425][2nd=2977]}) [2025-05-31 02:58:49.978923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.985048] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:49.989623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:49.995266] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:50.001412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.012222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.022532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.033251] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.040557] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB94681-0-0] [lt=13][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:1328410521, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678330066210, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1381981, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678330052102, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678330040552, enqueue_timestamp_:0, request_arrival_time_:1748678330040552, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:50.040676] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB94681-0-0] [lt=100][errcode=-5150] deliver rpc request fail(&req=0x2b099f604098, ret=-5150) [2025-05-31 02:58:50.045001] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:338) [83079][T1_L0_G0][T1][YB42AC1087A7-00063665444DFCE5-0-0] [lt=54][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:11577709}, ref_cnt:1, state:"RUNNING", commit_version:{val:1748678330040944001, v:0}, start_scn:{val:1748678330040944001, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1748678359942704, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678330040944001, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, data_trans_id_:{txid:11577709}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1748678330040944001, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1748678330040944001, v:0}}) [2025-05-31 02:58:50.045622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.052628] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-05-31 02:58:50.052731] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:58:50.057352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.064996] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:50.066954] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:50.067600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.072893] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:50.072950] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=6213, avg_group_log_size=443) [2025-05-31 02:58:50.072973] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5746) [2025-05-31 02:58:50.072988] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:50.075372] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=3381, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:50.075400] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=6213) [2025-05-31 02:58:50.075432] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=471, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:50.075507] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=275, avg_log_gen_to_submit_time=361, avg_log_submit_to_flush_time=4308, avg_log_submit_to_slide_time=4548) [2025-05-31 02:58:50.075542] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=61, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:50.075633] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=6213, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:50.080434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.083168] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:50.090818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.094185] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81740][T1_TimerWK0_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=110] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:50.097926] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A7-00063665469F7B51-0-0] [lt=90] get number(ret=0, service_type_=0, range=1, base_id=1748678330096892000, start_id=1748678330096892000, end_id=1748678330096892001) [2025-05-31 02:58:50.101099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.103322] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2066-0-0] [lt=236] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:50.103694] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2066-0-0] [lt=361] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=374) [2025-05-31 02:58:50.111361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.118914] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:50.119879] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A383F-0-0] [lt=146][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525379302048, cache_obj->added_lc()=false, cache_obj->get_object_id()=53618, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:50.124521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.132050] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:50.134986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.145232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.155480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.167358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.167651] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=39] ObArchiveSender is running(thread_index=0) [2025-05-31 02:58:50.177562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.188156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.200826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.211186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.223380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.233633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.242463] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=55] new mysql sessid created(conn.sessid_=3221736896, support_ssl=false) [2025-05-31 02:58:50.242546] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] sm conn init succ(conn.sessid_=3221736896, sess.client_addr_="172.16.135.166:43876") [2025-05-31 02:58:50.242567] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x2b08ff896450, session_id:3221736896, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:141}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.242989] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81568][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:58:50.243009] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [81568][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=19] mysql login queue(mysql_queue->queue_.size()=1) [2025-05-31 02:58:50.243089] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCC-0-0] [lt=26] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221736896, conn.client_addr_port_=43876, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221736896, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221736896, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-0006366544F9FFCC-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82989, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418611164674781, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:94, fixup_index:-1, fixup_ash_buffer:{this:0x2b09a0def198, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:58:50.245087] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCC-0-0] [lt=65][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:58:50.245181] INFO [SERVER] process (obmp_connect.cpp:504) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCC-0-0] [lt=21] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221736896, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:58:50.245596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.248642] INFO [SERVER] process (obmp_init_db.cpp:44) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C7A-0-0] [lt=32] init db(db_name=oceanbase) [2025-05-31 02:58:50.259682] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA6F-0-0] [lt=41][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525379441790, cache_obj->added_lc()=false, cache_obj->get_object_id()=53619, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:50.260344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.273031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.273864] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:50.276934] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221736897, support_ssl=false) [2025-05-31 02:58:50.276988] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221736897, sess.client_addr_="172.16.135.166:43880") [2025-05-31 02:58:50.277063] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=69] accept one succ(*s={this:0x2b08ff8971d0, session_id:3221736897, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.278233] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81568][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:58:50.278460] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE5-0-0] [lt=59] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221736897, conn.client_addr_port_=43880, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221736897, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221736897, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-000636654349FFE5-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82987, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418611259817540, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:235, fixup_index:-1, fixup_ash_buffer:{this:0x2b09a0ded6f8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:58:50.280642] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE5-0-0] [lt=294][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:58:50.280782] INFO [SERVER] process (obmp_connect.cpp:504) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE5-0-0] [lt=18] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221736897, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:58:50.281773] INFO [SERVER] process (obmp_init_db.cpp:44) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C7D-0-0] [lt=25] init db(db_name=oceanbase) [2025-05-31 02:58:50.283277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.284822] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE41A9-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1748678330284776000, start_id=1748678330284776000, end_id=1748678330284776001) [2025-05-31 02:58:50.290413] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C80-0-0] [lt=65][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:50.293543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.303752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.308678] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=29] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:50.308705] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:50.308770] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3E-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:50.308790] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3E-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:50.313949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.319125] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:50.326119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.332752] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:50.338817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.339096] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221736916, support_ssl=false) [2025-05-31 02:58:50.339152] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221736916, sess.client_addr_="172.16.135.166:43884") [2025-05-31 02:58:50.339170] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2b08ff9f2bd0, session_id:3221736916, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:153}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.339747] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81568][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:58:50.340134] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDF-0-0] [lt=27] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221736916, conn.client_addr_port_=43884, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221736916, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221736916, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-000636654429FFDF-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82988, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418611425939888, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:394, fixup_index:-1, fixup_ash_buffer:{this:0x2b09a0debc58, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:58:50.342035] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDF-0-0] [lt=72][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:58:50.342199] INFO [SERVER] process (obmp_connect.cpp:504) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFDF-0-0] [lt=91] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221736916, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:58:50.343255] INFO [SERVER] process (obmp_init_db.cpp:44) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C81-0-0] [lt=44] init db(db_name=oceanbase) [2025-05-31 02:58:50.349231] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA69-0-0] [lt=91][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:50.350780] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83078][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C83-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:50.350964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.359907] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.359972] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.360054] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.360085] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.361975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.366644] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=93] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:50.373969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.384209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.395961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.397161] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=1993, gc_interval=10000000) [2025-05-31 02:58:50.397283] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=95] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:58:50.397308] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=15] gc_check_member_list_ cost time(ret=0, time_us=51) [2025-05-31 02:58:50.397321] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:58:50.397333] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:58:50.397346] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:58:50.397356] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=12) [2025-05-31 02:58:50.397362] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] ObSafeDestroyHandler start process [2025-05-31 02:58:50.397392] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=27] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:58:50.397414] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:58:50.406328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.412183] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=99] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=756, get_gts_with_stc_cnt=32, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=32, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:50.412216] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678330412081], gts:1748678330410808000, latest_srr:[mts=1748678330412081]}) [2025-05-31 02:58:50.416681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.426426] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3841-0-0] [lt=111][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525379608595, cache_obj->added_lc()=false, cache_obj->get_object_id()=53620, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:50.427801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.438189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.446008] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81064][TimerWK2_AutoLSLocRpc][T0][YB42AC1087A6-0006366542DA545F-0-0] [lt=22] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:58:50.446659] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3843-0-0] [lt=27][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:50.447701] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3845-0-0] [lt=77][errcode=-4201] hashtable not init [2025-05-31 02:58:50.447980] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83082][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1748678330446496000, start_id=1748678330446496000, end_id=1748678330446496001) [2025-05-31 02:58:50.449901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.461306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.471658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.484661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.485455] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC108714-000636671E6D512A-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525379667624, cache_obj->added_lc()=false, cache_obj->get_object_id()=53621, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:50.486546] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=175] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:50.487707] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137220880}], scn_array=[cnt:1, {val:1748678330471757000, v:0}], curr_size=589, accum_size=1866, time_cost=10435) [2025-05-31 02:58:50.487914] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=22, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:50.487938] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=5530, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:50.488234] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678330069237001, v:0}, new_max_version={val:1748678330069237001, v:0}, last_min_version={val:1748678330040944001, v:0}, last_max_version={val:1748678330040944001, v:0}, total_time_us=13463, delta_us=418994) [2025-05-31 02:58:50.488269] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678330069237001, v:0}, min_version={val:1748678330069237001, v:0}, max_version={val:1748678330069237001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:50.488315] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=219664, log_service_ts=1748678330268457, min_tx_service_ts=4611686018427387, timestamp={val:1748678330268457000, v:0}) [2025-05-31 02:58:50.488334] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=59, Other_Error_Cnt=0, Submit_Succ_Cnt=40, last_scn="{val:1748678325630415001, v:0}", last_lsn={lsn:2137200297}, last_gts={val:1748678330446496000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:58:50.495187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.497277] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:50.505393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.511371] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff8971d0, session_id:3221736897, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678330284776, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1051, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.511443] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=59] kill and revert session(conn.sessid_=3221736897, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:58:50.511458] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff896450, session_id:3221736896, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678330249837, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.511477] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221736896, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:58:50.511485] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sock ref clean, do destroy(*s={this:0x2b08ff8971d0, session_id:3221736897, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678330284776, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1051, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.511540] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] unlock session of tenant(conn.sessid_=3221736897, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:58:50.511554] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221736897, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654CCA1C80-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:11, cur_pkt_pos_:17, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:9835554, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:9835555, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:9835556, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:9835556, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:9835556, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:9835556, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:9835556, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:16}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:9835556, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:17}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:9835556, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:18}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:194, pkt_seq:0, req_id:9835557, flag_.flags:30, ext_flags:0}, obp_mysql_header_{len_:190, rec_:194, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:11}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:12}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:738, com_seq:1}, obp20_header_:{payload_len:710, pkt_seq:1, req_id:9835557, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:13}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:58:50.511708] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=148] sock ref clean, do destroy(*s={this:0x2b08ff896450, session_id:3221736896, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678330249837, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:58:50.511756] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221736896, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:58:50.511768] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221736896, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654CCA1C7C-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:1179572, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:1179573, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:1179574, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1179574, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1179574, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1179574, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:1179574, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:1179574, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:1179574, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:58:50.511928] INFO [SERVER] run (obmp_disconnect.cpp:78) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=24] free session successfully(sessid=3221736897) [2025-05-31 02:58:50.511948] INFO [SERVER] run (obmp_disconnect.cpp:85) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=18] mark session id unused(sessid=3221736897) [2025-05-31 02:58:50.512017] INFO [SERVER] run (obmp_disconnect.cpp:78) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=6] free session successfully(sessid=3221736896) [2025-05-31 02:58:50.512029] INFO [SERVER] run (obmp_disconnect.cpp:85) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=11] mark session id unused(sessid=3221736896) [2025-05-31 02:58:50.516186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.519223] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:50.524651] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] ====== checkpoint timer task ====== [2025-05-31 02:58:50.525712] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:50.525759] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=46] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:50.526352] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:50.526654] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=296] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:50.526722] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:50.526739] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1748678323711104001, v:0}) [2025-05-31 02:58:50.526748] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:50.526756] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:50.526770] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:50.526790] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:50.526804] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:50.526824] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:50.526846] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:50.526917] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=68] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:50.527515] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.533251] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:50.537893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.548244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=155][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.558546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.569642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.574816] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CB-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:50.580037] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.591048] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.591096] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.591178] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.591241] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.591250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.591271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.591297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.593051] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:50.601441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.611815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.622365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.634599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.644934] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.658511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.669295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.673793] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5398) [2025-05-31 02:58:50.677440] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9B8-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.652903938293457031e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.723171724460813493e+03) [2025-05-31 02:58:50.678726] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:933223 disk:48K/11:933223 t1m2:48K/11:933223 t1m2g0:48K/11:933223 |q0: 0 |q1: 0 [2025-05-31 02:58:50.680341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.683516] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:50.690825] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.701682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=141][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.712310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.720765] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:50.725278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.729307] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:50.729364] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=47] temporary file module use no blocks [2025-05-31 02:58:50.729377] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:50.729392] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:50.729407] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525378908670, flush_io_finished_ret_:0, flush_io_finished_round_:19898, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:50.734482] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:50.735791] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.747573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.758375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.768828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.772185] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:50.772213] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] try gc retain ctx [2025-05-31 02:58:50.772221] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-31 02:58:50.772415] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1748678330677627000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678320473655001, v:0}, keep_alive_scn_:{val:1748678325630415001, v:0}, update_ts_:1748678330771206}) [2025-05-31 02:58:50.773134] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:50.779477] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.790922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.801175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.805296] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447B-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:50.808286] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=8] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137222293}, readable_log_tail:{lsn:2137222293}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10130283, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:50.808404] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=117] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:58:50.808442] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=15] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:58:50.811380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.821595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.830652] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=33] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032161, max_lsn:2137222293, max_scn:{val:1748678330677627001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137222293}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095279253}}, last_submit_lsn:{lsn:2137222180}, last_submit_end_lsn:{lsn:2137222293}, last_submit_log_id:4032161, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137222180}, max_flushed_end_lsn:{lsn:2137222293}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137222293}, last_slide_log_id:4032161, last_slide_scn:{val:1748678330677627001, v:0}, last_slide_lsn:{lsn:2137222180}, last_slide_end_lsn:{lsn:2137222293}, last_slide_log_pid:11, last_slide_log_accum_checksum:3448633977, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032162, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137222293}, readable_log_tail:{lsn:2137222293}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10130283, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:58:50.831106] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=444] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678330777125, last_flushed_end_lsn:{lsn:2137222293}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678330774720, last_flushed_end_lsn:{lsn:2137222293}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678330780215, last_flushed_end_lsn:{lsn:2137222293}}]) [2025-05-31 02:58:50.831133] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032162, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678330777125, last_flushed_end_lsn:{lsn:2137222293}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678330774720, last_flushed_end_lsn:{lsn:2137222293}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678330780215, last_flushed_end_lsn:{lsn:2137222293}}]) [2025-05-31 02:58:50.831348] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:58:50.831378] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=21] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:50.831399] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:50.831884] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.842086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.852285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.862610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.872809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.879074] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:50.884494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.884552] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:50.887339] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [10191][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=9] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-05-31 02:58:50.894931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.905744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.915068] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:50.915733] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=62] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:50.915974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.918974] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=36] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:50.921026] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=61] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:50.926332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.936356] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678330936352, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000026, add_bytes: 0 [2025-05-31 02:58:50.936876] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:50.938205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.949165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.959829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.971074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.980821] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:50.980987] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=99] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48911, [delay/us]:prepare:20, schedule:354, submit:223, rt:2892, total:3490) [2025-05-31 02:58:50.981009] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:50.981096] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=55] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48915, [delay/us]: prepare:20, schedule:354, submit:223, device:2892, total:3490) [2025-05-31 02:58:50.981111] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:50.981138] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4076, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:454, total:469) [2025-05-31 02:58:50.981186] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:50.981278] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4076, [delay/us]: prepare:0, schedule:0, submit:600000000, device:454, total:469) [2025-05-31 02:58:50.981299] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:50.981315] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:50.981323] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:50.981331] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:50.981340] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:50.981349] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=163kB/s, net_out=103kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:50.981370] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399426][2nd=2554]}) [2025-05-31 02:58:50.982797] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1748678330982793, bytes: 7130867477, bw: 0.158714 MB/s, add_ts: 1009805, add_bytes: 168056 [2025-05-31 02:58:50.983209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.986697] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:50.993633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:50.997421] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:51.003859] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.014047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.018740] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [19591][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9C-0-0] [lt=332] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678331051995, keepalive_ts:-1}) [2025-05-31 02:58:51.018814] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [19591][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9C-0-0] [lt=71] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678331051995, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:51.026764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.030971] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2025-05-31 02:58:51.031001] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:51.032051] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=29] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:58:51.036942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.047248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.053430] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=19] gc stale ls task succ [2025-05-31 02:58:51.057470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.065582] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=214] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:51.068362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.079667] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=123] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:51.079710] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3522, avg_group_log_size=352) [2025-05-31 02:58:51.079778] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3989) [2025-05-31 02:58:51.079796] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:51.083343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.085261] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=46] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:51.086257] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4172, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:51.086285] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3522) [2025-05-31 02:58:51.086322] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=998, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:51.089154] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=581, avg_log_gen_to_submit_time=676, avg_log_submit_to_flush_time=5497, avg_log_submit_to_slide_time=6469) [2025-05-31 02:58:51.089212] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=74, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:51.090484] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3522, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:51.093773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.104198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.114424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.121201] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:51.131964] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3847-0-0] [lt=17][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525380314132, cache_obj->added_lc()=false, cache_obj->get_object_id()=53622, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:51.132151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.137361] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:51.138636] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=723511, clean_num=31457) [2025-05-31 02:58:51.142771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.152986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.160947] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:51.161014] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=53] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:51.161099] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81742][T1_TimerWK1_][T1][Y0-0000000000000000-0-0] [lt=81] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1011) [2025-05-31 02:58:51.163806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.168291] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:51.174096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.186273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.196405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.207147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.217322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.226683] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23] === Run print tenant memory usage task === [2025-05-31 02:58:51.226784] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=74] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 120,934,400 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 649,641,984 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,331,226,603 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 52,809,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 895,873,024 max_mem_memstore_can_get_now= 264,046,576 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:58:51.227277] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,071,053,824 total_hold= 3,321,888,768 used= 1,574,588,416 freelists_hold= 1,496,465,408 total_maps= 2,017 total_unmaps= 806 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 120,934,400 virtual_memory_used= 4,342,378,496 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,002,438,656 count= 478 pushes= 14,520,155 pops= 14,519,677 maps= 1,822 unmaps= 762 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,938 pops= 22,934 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:58:51.227308] INFO print (ob_malloc_time_monitor.cpp:37) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 8100, delta_count= 3, avg_cost_time= 2700 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:58:51.227840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.238586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.249776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.260365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.270641] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.281105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=171][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.292664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.304261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.314863] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.326237] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:51.331134] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=51] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:51.332666] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=45] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:51.332767] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:51.332805] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=35] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678322139457000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678322139457000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:58:51.333233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=150][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.338996] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:51.342554] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:51.343529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.345961] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=54] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:51.348023] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=28] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:51.350801] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=20] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:51.350977] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=173] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:51.354757] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:51.354885] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=29] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:51.354904] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=17] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:51.354956] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=38] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:51.355462] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=501] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:51.355476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.355505] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=32] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:51.355516] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:51.355539] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:51.355559] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:51.357259] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:51.357312] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=48] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:51.357325] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:51.357363] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:51.357398] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=32] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:51.357409] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=11] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:51.357422] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:51.357438] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BD9-0-0] [lt=14] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:51.359690] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.359733] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.361132] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.361189] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.365974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.377007] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA6A-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:51.377758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.388404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.399730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.411331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.421738] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3849-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525380603905, cache_obj->added_lc()=false, cache_obj->get_object_id()=53623, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:51.421949] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=51] ts response task statistics(alloc_count=486098, free_count=243048) [2025-05-31 02:58:51.422041] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678330915133], gts:1748678330949979000, latest_srr:[mts=1748678331422001]}) [2025-05-31 02:58:51.422693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.435113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.446197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.456705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.467456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.477770] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [17207][T1_L0_G19][T1][YB42AC10871E-00063665511850D7-0-0] [lt=46] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:58:51.477823] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [17207][T1_L0_G19][T1][YB42AC10871E-00063665511850D7-0-0] [lt=41] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:58:51.480601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.490503] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=48] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-31 02:58:51.491152] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.497957] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:51.502718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.513125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.523505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.526963] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:51.535106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.543451] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:51.548090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.558540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.568899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.579648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.587757] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:51.587791] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6340, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:51.590026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.591586] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.591785] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.591911] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.591947] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.591973] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.591998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.592030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:51.600533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.610963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.621247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.622718] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A6A-0-0] [lt=30] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:51.634367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.644546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.657331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.666979] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81740][T1_TimerWK0_Flush][T1][Y0-0000000000000000-0-0] [lt=31] ====== traversal_flush timer task ====== [2025-05-31 02:58:51.667695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.667518] INFO [MDS] flush (mds_table_mgr.cpp:184) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F19-0-0] [lt=27] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678331477741000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678340364653000, latest_log_ts:{val:1748678323711104001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:51.667804] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F19-0-0] [lt=277] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:51.678020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.680338] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9C3-0-0] [lt=56] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.725054693222045898e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.589602999672082206e+03) [2025-05-31 02:58:51.682402] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=33] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:297713 disk:48K/11:297713 t1m2:48K/11:297713 t1m2g0:48K/11:297713 |q0: 0 |q1: 0 [2025-05-31 02:58:51.684237] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [15659][T1_L0_G1][T1][YB42AC10871E-0006366549CC10AA-0-0] [lt=30] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3667) [2025-05-31 02:58:51.688583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.690024] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=56981424, this={dio_aligned_buf:{buf_write_offset:2557, buf_padding_size:1539, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:390013, truncate_used_ts:132150}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=56979456, buf_len=589, total_write_size=54475, total_write_size_after_dio=573440, ob_pwrite_used_ts=428140, count=127, each_pwrite_cost=3371) [2025-05-31 02:58:51.693249] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678331280072001, v:0}, new_max_version={val:1748678331280072001, v:0}, last_min_version={val:1748678331076726000, v:0}, last_max_version={val:1748678331076726000, v:0}, total_time_us=14567, delta_us=413173) [2025-05-31 02:58:51.693304] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678331280072001, v:0}, min_version={val:1748678331280072001, v:0}, max_version={val:1748678331280072001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:51.695283] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:51.697059] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F1A-0-0] [lt=20] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=32) [2025-05-31 02:58:51.697274] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678331692926000, start_id=1748678331692926000, end_id=1748678331692926001) [2025-05-31 02:58:51.697519] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3D8-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525380879691, cache_obj->added_lc()=false, cache_obj->get_object_id()=53624, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:51.699632] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.710558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.720961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.727230] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:51.729674] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:51.729788] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=105] temporary file module use no blocks [2025-05-31 02:58:51.729892] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=101] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:51.729915] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:51.730024] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=108] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525379911622, flush_io_finished_ret_:0, flush_io_finished_round_:19899, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:51.733675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.743883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.744089] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:51.754376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.760548] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=84] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:51.760581] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:51.760594] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:51.760605] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:51.763580] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:51.764389] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81743][T1_TimerWK2_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=803] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:51.764695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.775096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.786951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.797221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.808663] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447C-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:51.809389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.819703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.830444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.840801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.851228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.862105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.872528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.882833] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.892124] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:51.893242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.893296] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:51.894245] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [15659][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=30] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-05-31 02:58:51.903639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.914851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.919644] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=60] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:51.923020] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=107] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:51.923305] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678331422467], gts:1748678331458768000, latest_srr:[mts=1748678331923287]}) [2025-05-31 02:58:51.925150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.928395] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=251648, replace_num=15728) [2025-05-31 02:58:51.928428] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:51.935776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.942813] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678331942808, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006456, add_bytes: 0 [2025-05-31 02:58:51.946276] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.948340] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:51.959596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.970306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.981106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.987860] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1748678331987855, bytes: 7131079887, bw: 0.201550 MB/s, add_ts: 1005062, add_bytes: 212410 [2025-05-31 02:58:51.990688] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:51.990748] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, norm_iops:2, bw:40562, [delay/us]:prepare:30, schedule:464, submit:151, rt:2732, total:3378) [2025-05-31 02:58:51.990767] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:51.990827] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40565, [delay/us]: prepare:30, schedule:464, submit:151, device:2732, total:3378) [2025-05-31 02:58:51.990898] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:51.990930] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4056, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:293, total:308) [2025-05-31 02:58:51.990957] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:51.990990] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4056, [delay/us]: prepare:0, schedule:0, submit:600000000, device:293, total:308) [2025-05-31 02:58:51.991008] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:51.991022] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:51.991030] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:51.991039] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:51.991046] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:51.991055] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=205kB/s, net_out=113kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:51.991118] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399427][2nd=2836]}) [2025-05-31 02:58:51.991562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:51.991591] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:51.999110] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:52.001699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.012042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.022245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.032593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.043014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.053797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.054808] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=47] current monitor number(seq_=-1) [2025-05-31 02:58:52.055101] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:58:52.064181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.066202] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=47] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:52.074539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.084643] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=75, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:52.084701] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=4326, avg_group_log_size=480) [2025-05-31 02:58:52.084723] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3859) [2025-05-31 02:58:52.084739] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:52.085475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.085698] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:52.091607] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=3445, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:52.091636] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4326) [2025-05-31 02:58:52.091658] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=835, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:52.091728] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=912, avg_log_gen_to_submit_time=1007, avg_log_submit_to_flush_time=4333, avg_log_submit_to_slide_time=4764) [2025-05-31 02:58:52.091779] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=87, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:52.093116] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4326, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:52.095788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.106204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.108479] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2067-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:52.108516] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2067-0-0] [lt=28] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-05-31 02:58:52.116587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.129835] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.130975] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:52.140148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.141510] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83087][T1_L0_G0][T1][YB42AC1087A6-000636654C3A384B-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525381323669, cache_obj->added_lc()=false, cache_obj->get_object_id()=53625, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:52.147769] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33BF-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525381329941, cache_obj->added_lc()=false, cache_obj->get_object_id()=53626, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:52.148963] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:52.150700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.161094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.172344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.183824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=175][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:52.186122] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [20745][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:52.186323] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=176] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-05-31 02:58:52.197119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.199556] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AE8-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 03:58:52.201092] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [20750][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:52.201418] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:58:52.205718] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E4E-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1748678332204795000, start_id=1748678332204795000, end_id=1748678332204795001) [2025-05-31 02:58:52.213169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.226392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.236604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.246927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.255339] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=120] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:52.257154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.267398] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.277638] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.288031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.296685] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=34] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:52.298220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.310118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.316780] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:52.316823] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=41] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:52.317081] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3F-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:52.317133] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F3F-0-0] [lt=48] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:52.320233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.330372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.333107] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=99] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:52.340526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.349650] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:52.350704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.359208] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.359253] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=348] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.359967] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.360025] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.361548] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.372138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.383514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.394855] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.404011] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA6B-0-0] [lt=83][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:52.405200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.416971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.427198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.437885] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.448208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.458444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.468694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.478958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.486998] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get number(ret=0, service_type_=0, range=1, base_id=1748678332484309000, start_id=1748678332484309000, end_id=1748678332484309001) [2025-05-31 02:58:52.490963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.496056] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:52.499304] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:52.501193] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.507471] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137229495}], scn_array=[cnt:1, {val:1748678332484309000, v:0}], curr_size=589, accum_size=2633, time_cost=12131) [2025-05-31 02:58:52.514100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.524549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.534986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.541666] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:52.546991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.553214] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:52.557182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.568969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.582544] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CC-0-0] [lt=38] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:52.585678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.585945] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] get_max_applied_scn(scn={val:1748678332484309000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137230084}, palf_committed_end_scn_:{val:1748678332484309000, v:0}, last_check_scn_:{val:1748678332484309000, v:0}, max_applied_cb_scn_:{val:1748678332484309000, v:0}}) [2025-05-31 02:58:52.585983] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137230084}, readable_log_tail:{lsn:2137230084}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10122492, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:52.588369] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A564E-0-0] [lt=31] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:52.590940] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.591191] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.591228] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.591304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.591331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.591357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.592003] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:52.595981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.599454] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:52.599562] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=106] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7506, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:52.600232] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:52.600437] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=192][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:52.600480] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:52.600503] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:52.601423] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] ls blacklist refresh finish(cost_time=31647) [2025-05-31 02:58:52.606265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.616728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.627019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.637395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.647596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.657736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.668077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.678278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.689354] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9D2-0-0] [lt=82] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701055288314819336e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633103737399523652e+03) [2025-05-31 02:58:52.690435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.690978] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:570925 disk:44K/10:570925 t1m2:44K/10:570925 t1m2g0:44K/10:570925 |q0: 0 |q1: 0 [2025-05-31 02:58:52.692738] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [19591][T1_L0_G1][T1][YB42AC10871E-0006366549CC10B4-0-0] [lt=81] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5070) [2025-05-31 02:58:52.694212] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678332484309000, v:0}, new_max_version={val:1748678332484309000, v:0}, last_min_version={val:1748678332179392000, v:0}, last_max_version={val:1748678332179392000, v:0}, total_time_us=7732, delta_us=209900) [2025-05-31 02:58:52.694307] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678332484309000, v:0}, min_version={val:1748678332484309000, v:0}, max_version={val:1748678332484309000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:52.701270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.711740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.721929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.730273] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=87] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:52.730316] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-31 02:58:52.730327] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:52.730343] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:52.730358] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525380912296, flush_io_finished_ret_:0, flush_io_finished_round_:19900, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:52.732392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.741797] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=23) [2025-05-31 02:58:52.742588] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:52.742624] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:58:52.743368] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] update_check_range(min_check_version=1982, max_check_version=1983, global_version=1984) [2025-05-31 02:58:52.748729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.749226] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:52.754070] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [17207][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AD-0-0] [lt=99] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:58:52.754137] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [17207][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AD-0-0] [lt=54] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:58:52.754422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:52.759379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.769452] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 153 276 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 51 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 18 6 3 0 1 20 2 1 0 500 0 88 252 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 26130 [2025-05-31 02:58:52.769522] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=43] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 895,873,024 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 455,450,624 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 106,954,752 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 12,603,392 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:52.769944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.769986] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 455,450,624 used= 344,249,456 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 20,398,080 used= 20,177,472 count= 32 avg_used= 630,546 block_cnt= 32 chunk_cnt= 20 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,127,040 used= 6,258,688 count= 106 avg_used= 59,044 block_cnt= 106 chunk_cnt= 29 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,818,048 used= 2,380,480 count= 86 avg_used= 27,680 block_cnt= 86 chunk_cnt= 26 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 966,656 used= 474,832 count= 59 avg_used= 8,048 block_cnt= 59 chunk_cnt= 21 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 397,824 used= 381,696 count= 84 avg_used= 4,544 block_cnt= 84 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 294,656 used= 283,760 count= 6 avg_used= 47,293 block_cnt= 6 chunk_cnt= 4 mod=CommSysVarFac [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 100,208 used= 26,000 count= 368 avg_used= 70 block_cnt= 100 chunk_cnt= 28 mod=Coro [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 54,432 used= 42,336 count= 63 avg_used= 672 block_cnt= 58 chunk_cnt= 23 mod=[T]ObWarningBuf [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 47,360 used= 33,152 count= 74 avg_used= 448 block_cnt= 44 chunk_cnt= 13 mod=OMT_Worker [MEMORY] hold= 45,008 used= 4,800 count= 200 avg_used= 24 block_cnt= 74 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 3 mod=SqlSession [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=SqlSessiVarMap [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PlanVaIdx [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,880 used= 1,920 count= 5 avg_used= 384 block_cnt= 5 chunk_cnt= 4 mod=TableLockNode [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,424 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 2 mod=SqlSessiQuerSql [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=ContextsMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceIdMap [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceMap [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=PackStateMap [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 344,249,456 used= 336,216,427 count= 32,598 avg_used= 10,314 mod=SUMMARY [2025-05-31 02:58:52.770082] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=77] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:58:52.770103] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:58:52.770125] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,574,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,511,808 used= 1,476,096 count= 186 avg_used= 7,936 block_cnt= 186 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,360,000 used= 1,240,000 count= 625 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,574,720 used= 35,241,449 count= 3,357 avg_used= 10,497 mod=SUMMARY [2025-05-31 02:58:52.770144] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:58:52.770157] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 106,954,752 used= 104,767,488 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 104,767,488 used= 104,572,608 count= 203 avg_used= 515,136 block_cnt= 203 chunk_cnt= 51 mod=CoStack [MEMORY] hold= 104,767,488 used= 104,572,608 count= 203 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:58:52.770176] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:58:52.770192] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:58:52.770229] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:52.770245] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:52.770258] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:58:52.770309] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 12,603,392 used= 8,969,664 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,779,008 used= 5,642,496 count= 711 avg_used= 7,936 block_cnt= 711 chunk_cnt= 4 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 8,969,664 used= 8,804,160 count= 715 avg_used= 12,313 mod=SUMMARY [2025-05-31 02:58:52.770327] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:58:52.770394] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 649,641,984 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:58:52.770732] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 180,841,456 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT value FROM __all_zone where zone=? and name=?) [2025-05-31 02:58:52.811363] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447D-0-0] [lt=20][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:52.815811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.827792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.835027] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [19591][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3A-0-0] [lt=161] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678332834053, keepalive_ts:-1}) [2025-05-31 02:58:52.835065] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [19591][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3A-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678332834053, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:52.837989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.848624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.854779] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:58:52.854896] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=106] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:52.854924] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=18] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:52.859329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.870574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.882527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.893552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.906442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.916952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.920687] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:52.924482] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1748678332923618000, start_id=1748678332923618000, end_id=1748678332923618001) [2025-05-31 02:58:52.924540] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:52.929491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.943705] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=130] PNIO [ratelimit] time: 1748678332943701, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000893, add_bytes: 0 [2025-05-31 02:58:52.944161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.945848] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=48] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:52.949404] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:52.954381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.955117] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=77] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:52.959404] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83104][T1_L0_G0][T1][YB42AC1087A7-0006366543FE060C-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678332957812000, start_id=1748678332957812000, end_id=1748678332957812001) [2025-05-31 02:58:52.965500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.966151] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=41, stat_interval=5000000, avg (count/cost)=156, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:52.966193] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=41] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=41, stat_interval=5000000, avg (count/cost)=6189, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:52.966204] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=41, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:52.966212] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=41, stat_interval=5000000, avg (count/cost)=-1066, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:52.966218] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=41, stat_interval=5000000, avg (count/cost)=5278, this=0x2b0913fa6b40, extra_info="") [2025-05-31 03:58:52.967946] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [20745][T1_L0_G1][T1][YB42AC1087A7-00063665517F54F1-0-0] [lt=44] init point thread id with(&point=0x55c0908cb600, idx_=265, point=[thread id=20745, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=20745) [2025-05-31 02:58:52.976450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.987092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.989333] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678332989329, bytes: 7131271572, bw: 0.182536 MB/s, add_ts: 1001474, add_bytes: 191685 [2025-05-31 02:58:52.996508] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=104] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:52.997323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:52.999921] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:52.999994] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, norm_iops:3, bw:52760, [delay/us]:prepare:17, schedule:493, submit:126, rt:3559, total:4196) [2025-05-31 02:58:53.000013] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:53.000070] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:52760, [delay/us]: prepare:17, schedule:493, submit:126, device:3559, total:4196) [2025-05-31 02:58:53.000084] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:53.000110] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4058, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:612, total:627) [2025-05-31 02:58:53.000135] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:53.000163] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4058, [delay/us]: prepare:0, schedule:0, submit:600000000, device:612, total:627) [2025-05-31 02:58:53.000181] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:53.000195] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:53.000203] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:53.000211] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:53.000220] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:53.000228] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=185kB/s, net_out=156kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:53.000249] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399428][2nd=2413]}) [2025-05-31 02:58:53.001662] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=94] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:53.007456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.018080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.028347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.031215] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=11] ====== check clog disk timer task ====== [2025-05-31 02:58:53.031245] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:53.040016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.050206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.055114] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=99] current monitor number(seq_=-1) [2025-05-31 02:58:53.055273] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=23] gc stale ls task succ [2025-05-31 02:58:53.060464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.067926] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:53.070807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.081038] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.086358] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=17] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:53.091263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.092664] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:53.092718] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=5192, avg_group_log_size=472) [2025-05-31 02:58:53.092741] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5192) [2025-05-31 02:58:53.092755] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:53.100586] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=126] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4526, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:53.100619] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5192) [2025-05-31 02:58:53.100643] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1378, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:53.100701] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=104, avg_log_submit_to_flush_time=6139, avg_log_submit_to_slide_time=6576) [2025-05-31 02:58:53.100761] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=104, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:53.101058] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:53.102100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.102552] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5192, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:53.112966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.125207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.138675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.150998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.151079] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A384D-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525382333197, cache_obj->added_lc()=false, cache_obj->get_object_id()=53627, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:53.151414] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:53.155761] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=97] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:53.157262] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=754968, clean_num=31457) [2025-05-31 02:58:53.162187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.172461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.183334] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.193599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.193651] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:53.197189] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [16802][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=19] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-05-31 02:58:53.203910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.204014] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:58:53.205708] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [20750][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=3317] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:58:53.215464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.225801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.238108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=244][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.248416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.261027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.271255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.281590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.284712] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=99] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:53.291826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.302117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.306562] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=16] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:53.313062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.323365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.333776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.344009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.351643] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:53.355062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.358493] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:53.358938] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.358998] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.360204] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.360243] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.365885] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.368342] INFO pn_ratelimit (group.c:653) [81065][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=26] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:53.376536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.386812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.397093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.409299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.421664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.425753] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678333425704], gts:1748678333424454000, latest_srr:[mts=1748678333425704]}) [2025-05-31 02:58:53.431052] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA6C-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:53.431952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.442643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.447634] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B29-0-0] [lt=29] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.452932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.463502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.474095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.484798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.495798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.497791] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:53.501791] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:53.506204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.519204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.529462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.539930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.550264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.552000] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:53.559098] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:53.560553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.570874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.581096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.590306] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2A-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.591041] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=49] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.591336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.591610] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.591683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.591712] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.591750] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.591776] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.592905] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:53.601588] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=38, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:53.601620] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5385, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:53.601774] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=156][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.607601] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A30-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525382789772, cache_obj->added_lc()=false, cache_obj->get_object_id()=53628, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:53.612424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.624799] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2B-0-0] [lt=32] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.624973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.635229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.645601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.655821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.666162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.671501] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=33] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:53.671561] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=41] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=964) [2025-05-31 02:58:53.677194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.684638] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2C-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.687357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.695475] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9E1-0-0] [lt=32] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701055288314819336e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.633103737399523652e+03) [2025-05-31 02:58:53.697780] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:762062 disk:44K/10:762062 t1m2:44K/10:762062 t1m2g0:44K/10:762062 |q0: 0 |q1: 0 [2025-05-31 02:58:53.697902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.698140] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [19591][T1_L0_G1][T1][YB42AC10871E-0006366549CC10BE-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3981) [2025-05-31 02:58:53.700197] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678333494620000, v:0}, new_max_version={val:1748678333494620000, v:0}, last_min_version={val:1748678333086926001, v:0}, last_max_version={val:1748678333086926001, v:0}, total_time_us=6911, delta_us=205574) [2025-05-31 02:58:53.700237] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678333494620000, v:0}, min_version={val:1748678333494620000, v:0}, max_version={val:1748678333494620000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:53.708403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.712004] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E57-0-0] [lt=117] submit log success(service_type_=0, last_id=1748678350364653000, limited_id=1748678350364653000) [2025-05-31 02:58:53.712033] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E57-0-0] [lt=28] submit log with lock success(service_type_=0, last_id=1748678350364653000, limited_id=1748678350364653000) [2025-05-31 02:58:53.717550] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1748678350364653000, latest_scn={val:1748678333710865001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:58:53.717689] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=137] handle submit callback(service_type_=0, success=true, used_ts=5687, log_ts={val:1748678333710865001, v:0}, limited_id=1748678350364653000) [2025-05-31 02:58:53.717707] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=16] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:58:53.717717] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3846-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:58:53.718582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:53.719217] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [21284][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:53.719350] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-05-31 02:58:53.729683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=205][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.730562] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:53.730600] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-05-31 02:58:53.730610] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:53.730626] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:53.730641] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525381912573, flush_io_finished_ret_:0, flush_io_finished_round_:19901, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:53.742578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=149][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.752133] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:53.752925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.762085] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:53.763270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.769701] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2D-0-0] [lt=44] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.776028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.786260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.796522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.807408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.814245] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447E-0-0] [lt=22][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:53.817682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.827941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.838178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.849430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.859953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.871628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.882363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.892580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.895326] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2E-0-0] [lt=30] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.905164] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.915758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.922769] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=52] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:53.926097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.927779] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:53.931283] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B2F-0-0] [lt=28] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:53.932770] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=22] XA scheduler heartbeat task statistics(avg_time=2) [2025-05-31 02:58:53.936441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.947177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.950376] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1748678333950371, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006670, add_bytes: 0 [2025-05-31 02:58:53.952538] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:53.959579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.962563] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:53.969991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.981946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.993913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:53.994912] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1748678333994907, bytes: 7131418506, bw: 0.139350 MB/s, add_ts: 1005578, add_bytes: 146934 [2025-05-31 02:58:53.998115] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:54.004505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.004654] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:54.006076] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:54.006175] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44781, [delay/us]:prepare:25, schedule:797, submit:659, rt:2095, total:3578) [2025-05-31 02:58:54.006197] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:54.006314] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=100] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44776, [delay/us]: prepare:25, schedule:797, submit:659, device:2095, total:3578) [2025-05-31 02:58:54.006329] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:54.006356] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4070, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:461, total:477) [2025-05-31 02:58:54.006382] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:54.006413] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4070, [delay/us]: prepare:0, schedule:0, submit:600000000, device:461, total:477) [2025-05-31 02:58:54.006481] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:54.006501] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:54.006520] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:54.006586] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:54.006620] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:54.006637] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=142kB/s, net_out=88kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:54.006681] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] successfully alloc block(macro_id={[ver=1,mode=0,seq=399429][2nd=2695]}) [2025-05-31 02:58:54.014752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.025056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.037691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.047932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.055462] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:58:54.055633] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=22] gc stale ls task succ [2025-05-31 02:58:54.058218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.069275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.069640] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=43] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:54.079806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.086976] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=44] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:54.090123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.099567] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=78, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:54.099663] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3094, avg_group_log_size=309) [2025-05-31 02:58:54.099690] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3561) [2025-05-31 02:58:54.099707] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:54.100335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.105522] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3751, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:54.105558] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3094) [2025-05-31 02:58:54.105598] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=503, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:54.106126] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=128, avg_log_submit_to_flush_time=4855, avg_log_submit_to_slide_time=5436) [2025-05-31 02:58:54.106243] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=90] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:54.109802] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=55] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3094, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:54.110800] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.115601] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2068-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:54.115640] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2068-0-0] [lt=30] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=41) [2025-05-31 02:58:54.121068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.131441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.144691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.153645] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=267376, replace_num=15728) [2025-05-31 02:58:54.153682] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:54.155053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.162182] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=11] finish to resize log external storage handler(current_ts=1748678334162176, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:58:54.162218] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=33] snapshot_traffic_info once [2025-05-31 02:58:54.163282] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A384F-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525383345452, cache_obj->added_lc()=false, cache_obj->get_object_id()=53629, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:54.166505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.167109] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:54.177025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.190254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.200652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.214009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.214176] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E5A-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678334212956000, start_id=1748678334212956000, end_id=1748678334212956001) [2025-05-31 02:58:54.224358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.236748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.247524] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.257940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.270189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.280589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.290819] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE41C1-0-0] [lt=8] get number(ret=0, service_type_=0, range=1, base_id=1748678334290280000, start_id=1748678334290280000, end_id=1748678334290280001) [2025-05-31 02:58:54.291126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.293957] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=107] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:54.298288] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=44] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:54.301731] INFO [STORAGE.TRANS] create_tx_ctx_ (ob_trans_ctx_mgr_v4.cpp:406) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9E8-0-0] [lt=15] transaction statistics(ls_id={id:1}, total_count=1) [2025-05-31 02:58:54.302984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.307420] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [21284][T1_L0_G1][T1][YB42AC10871E-0006366549CC10C5-0-0] [lt=26] init point thread id with(&point=0x55c0908d3cc0, idx_=804, point=[thread id=21284, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=21284) [2025-05-31 02:58:54.311047] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:54.313224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.318588] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=11] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:54.318615] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:54.318941] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F40-0-0] [lt=51] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:54.318970] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F40-0-0] [lt=25] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:54.324175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.334460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.339847] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=39] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:54.341353] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=103] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:54.341488] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=52] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678332169852000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:54.341527] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=36] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678332169852000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678332169852000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:58:54.345995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.353996] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:54.356184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.358981] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.359035] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.360932] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.360990] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.367430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.367622] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:54.377790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.388180] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.398525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.399065] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=131] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678334301528000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678334398977}, version_delta=97534) [2025-05-31 02:58:54.408700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.418923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.423023] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=104] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:58:54.428618] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=43] ts response task statistics(alloc_count=486170, free_count=243084) [2025-05-31 02:58:54.428725] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:54.428746] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678333927824], gts:1748678333962983000, latest_srr:[mts=1748678334428671]}) [2025-05-31 02:58:54.429385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.439591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.450083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=235][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.452403] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA6D-0-0] [lt=94][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:54.460422] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.462037] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83090][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3851-0-0] [lt=38][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525383644206, cache_obj->added_lc()=false, cache_obj->get_object_id()=53630, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:54.465040] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3853-0-0] [lt=45][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:54.466208] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3855-0-0] [lt=85][errcode=-4201] hashtable not init [2025-05-31 02:58:54.466400] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1748678334465970000, start_id=1748678334465970000, end_id=1748678334465970001) [2025-05-31 02:58:54.471241] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.482043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.492297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.498424] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:54.503993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.507532] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:54.507689] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B30-0-0] [lt=29] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:54.514314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.525528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.536245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.545295] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B31-0-0] [lt=42] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:58:54.546821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.548088] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:670) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=49] failed to get result(ret=-4008) [2025-05-31 02:58:54.548283] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:515) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=40] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748678246784954, start_date_:1748169746138009, last_date_:1748678246141150, this_date_:0, next_date_:1748679146138009, end_date_:64060560000000000, total_:239131641, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748613666964441, start_date_:1748613600000000, last_date_:1748613600006788, this_date_:0, next_date_:1749218400000000, end_date_:64060560000000000, total_:66957653, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748314024043185, start_date_:1748268000000000, last_date_:0, this_date_:0, next_date_:1748872800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748600847610427, start_date_:1748255246137533, last_date_:1748600846140058, this_date_:0, next_date_:1748687246137533, end_date_:64060560000000000, total_:3757845, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748168846136670, start_date_:1748700000000000, last_date_:0, this_date_:0, next_date_:1748700000000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748624400066065, start_date_:1748192400000000, last_date_:1748624400003705, this_date_:0, next_date_:1748710800000000, end_date_:64060560000000000, total_:347108, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:4, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748181773502269, start_date_:1748181600000000, last_date_:1748181600002317, this_date_:0, next_date_:1748786400000000, end_date_:64060560000000000, total_:173499952, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748527281005724, start_date_:1748527200000000, last_date_:1748527200011176, this_date_:0, next_date_:1749132000000000, end_date_:64060560000000000, total_:80994548, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748437312134789, start_date_:1748354400000000, last_date_:0, this_date_:0, next_date_:1748959200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748440876223997, start_date_:1748440800000000, last_date_:1748440800003333, this_date_:0, next_date_:1749045600000000, end_date_:64060560000000000, total_:76220664, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}]) [2025-05-31 02:58:54.548519] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:504) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=231] check all tenants(ret=0) [2025-05-31 02:58:54.554208] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:54.557351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.566277] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F8E3F1D-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678334565051000, start_id=1748678334565051000, end_id=1748678334565051001) [2025-05-31 02:58:54.567593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.568177] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:54.580366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.582188] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137238191}], scn_array=[cnt:1, {val:1748678334565051000, v:0}], curr_size=548, accum_size=3002, time_cost=15639) [2025-05-31 02:58:54.590402] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CD-0-0] [lt=21] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:54.593065] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593171] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593231] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593279] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.593306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:54.595089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.604184] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A6B-0-0] [lt=53] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:54.606788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.617310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.632450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.645806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.657413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.668539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.678757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.689114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=152][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.697046] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [19591][T1_L0_G1][T1][YB42AC10871E-0006366549CC10C9-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4552) [2025-05-31 02:58:54.701420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.702035] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=15, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:54.702065] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=7300, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:54.713401] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83101][T1_L0_G0][T1][YB42AC10871E-000636654378A50E-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678334710287000, start_id=1748678334710287000, end_id=1748678334710287001) [2025-05-31 02:58:54.714791] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=77] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:56K/12:1334360 disk:56K/12:1334360 t1m2:56K/12:1334360 t1m2g0:56K/12:1334360 |q0: 0 |q1: 0 [2025-05-31 02:58:54.718283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.725590] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678334465970000, v:0}, new_max_version={val:1748678334465970000, v:0}, last_min_version={val:1748678334301528000, v:0}, last_max_version={val:1748678334301528000, v:0}, total_time_us=10962, delta_us=259616) [2025-05-31 02:58:54.725652] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678334465970000, v:0}, min_version={val:1748678334465970000, v:0}, max_version={val:1748678334465970000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:54.730298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.730362] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:54.730831] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:54.730921] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=81] temporary file module use no blocks [2025-05-31 02:58:54.730933] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:54.730949] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:54.730963] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525382912854, flush_io_finished_ret_:0, flush_io_finished_round_:19902, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:54.735055] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [20745][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-05-31 02:58:54.740910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.751213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.754539] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:54.761452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.772843] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:54.773119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.783591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.794116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.804262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.814898] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.817504] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A447F-0-0] [lt=84][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:54.817749] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678334565051000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=252686, in_cluster_service=true, cluster_version={val:1748678334465970000, v:0}, min_cluster_version={val:1748678334465970000, v:0}, max_cluster_version={val:1748678334465970000, v:0}, get_cluster_version_err=0, cluster_version_delta=351767, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195193, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=325762979, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:54.825438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.835680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.846269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.858137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.869056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.872677] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=89] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19337531392, fast_delete_log_mode=false) [2025-05-31 02:58:54.872974] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:54.873044] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:58:54.873072] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=18] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:54.880985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.891324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.903337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.915910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.919959] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9ED-0-0] [lt=42] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701206636428833008e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632826557530233913e+03) [2025-05-31 02:58:54.923530] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=29] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:54.926240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.930092] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:54.930494] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678334429142], gts:1748678334464873000, latest_srr:[mts=1748678334930423]}) [2025-05-31 02:58:54.936852] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.941118] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=42] success to update min reserved snapshot(reserved_snapshot=1748676532169852000, duration=1800, snapshot_gc_ts=1748678332169852000) [2025-05-31 02:58:54.947234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.947471] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=28] update reserved snapshot finished(cost_ts=6358, reserved_snapshot=1748676532169852000) [2025-05-31 02:58:54.951517] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1748678334951513, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001142, add_bytes: 0 [2025-05-31 02:58:54.954726] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=102] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:54.959959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.970827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.973672] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=76] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:54.982301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.994364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:54.999776] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-05-31 02:58:55.002464] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1748678335002460, bytes: 7131603653, bw: 0.175246 MB/s, add_ts: 1007553, add_bytes: 185147 [2025-05-31 02:58:55.007203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.008135] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:55.018176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.020487] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:55.020588] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56529, [delay/us]:prepare:20, schedule:1463, submit:70, rt:2819, total:4373) [2025-05-31 02:58:55.020611] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:55.020681] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56531, [delay/us]: prepare:20, schedule:1463, submit:70, device:2819, total:4373) [2025-05-31 02:58:55.020713] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:55.020797] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4037, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1666, total:1682) [2025-05-31 02:58:55.020955] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:55.021007] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4037, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1666, total:1682) [2025-05-31 02:58:55.021029] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:55.021047] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:55.021056] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:55.021065] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:55.021073] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:55.021083] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=178kB/s, net_out=108kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:55.021103] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399430][2nd=2567]}) [2025-05-31 02:58:55.028767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.031483] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-05-31 02:58:55.031517] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:55.036428] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [19591][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9D-0-0] [lt=39] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678335070187, keepalive_ts:-1}) [2025-05-31 02:58:55.036479] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [19591][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9D-0-0] [lt=47] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678335070187, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:55.039092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.049455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.053282] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83095][T1_L0_G0][T1][YB42AC10870A-000636671638EF33-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678335050107000, start_id=1748678335050107000, end_id=1748678335050107001) [2025-05-31 02:58:55.056164] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=40] current monitor number(seq_=-1) [2025-05-31 02:58:55.056294] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=23] gc stale ls task succ [2025-05-31 02:58:55.059676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.069029] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB946AE-0-0] [lt=12][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:3434270101, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678335097067, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1383049, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678335076252, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678335069024, enqueue_timestamp_:0, request_arrival_time_:1748678335069024, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:58:55.069122] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB946AE-0-0] [lt=74][errcode=-5150] deliver rpc request fail(&req=0x2b099d616098, ret=-5150) [2025-05-31 02:58:55.070195] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=42] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:55.070257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.080590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.088144] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:55.090892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.097397] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81740][T1_TimerWK0_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=41] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:58:55.102430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.112794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.118505] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678334916906000, v:0}, base_version={val:1748678334504294000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678334916906000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678335118470, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678334916906000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678335070299, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678334916906000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678335044494, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678330290426000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678335076323, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678330061167000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678335123372, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678330326759000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678335054981, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:58:55.121596] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=106, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:55.121629] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=6646, avg_group_log_size=474) [2025-05-31 02:58:55.121650] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=6179) [2025-05-31 02:58:55.121918] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:55.123333] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AEA-0-0] [lt=41] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:55.129515] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AEC-0-0] [lt=29] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:55.130208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.132333] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5306, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:55.132363] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137241794}], scn_array=[cnt:1, {val:1748678335120105000, v:0}], curr_size=589, accum_size=6646, time_cost=10194) [2025-05-31 02:58:55.132440] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=6646) [2025-05-31 02:58:55.132462] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=722, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:55.132598] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=691, avg_log_gen_to_submit_time=836, avg_log_submit_to_flush_time=6490, avg_log_submit_to_slide_time=6724) [2025-05-31 02:58:55.132652] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=123, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:55.134120] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=6646, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:55.142902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.154930] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:55.156416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.167818] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:58:55.169537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.174186] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:55.175828] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=786425, clean_num=31457) [2025-05-31 02:58:55.176259] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3857-0-0] [lt=18][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525384358429, cache_obj->added_lc()=false, cache_obj->get_object_id()=53631, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.179810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.190125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.200381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.213266] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.223501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.235489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.245802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.258055] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.268360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.281059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.295780] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA73-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525384477947, cache_obj->added_lc()=false, cache_obj->get_object_id()=53632, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.296159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.306648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.316956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.328315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.329444] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:55.339519] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.349742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.357136] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:55.358980] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.359038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.361750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.361814] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.362181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=153][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.376803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.377499] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:55.387327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.397715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.408055] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.418323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.430901] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=657, get_gts_with_stc_cnt=37, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=35, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:55.430926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.436672] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=49] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:55.441579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.452276] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.464986] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83097][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3859-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525384647153, cache_obj->added_lc()=false, cache_obj->get_object_id()=53633, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.465680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.476488] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA6E-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:55.478243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.488668] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.499119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.500221] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:55.508979] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:55.509403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.513702] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D512E-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525384695872, cache_obj->added_lc()=false, cache_obj->get_object_id()=53634, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.520791] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.523545] INFO [STORAGE.TRANS] get_tx_ctx_ (ob_trans_ctx_mgr_v4.cpp:483) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102] transaction statistics(ls_id={id:1}, total_tx_ctx_count=1) [2025-05-31 02:58:55.527619] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=29] ====== checkpoint timer task ====== [2025-05-31 02:58:55.528930] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:1, active_tx_count:1, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:58:55.528978] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=48] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:1748678335519417000, v:0}) [2025-05-31 02:58:55.529424] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:55.529519] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=89] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:58:55.529580] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=53] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:58:55.529595] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678333710865001, v:0}) [2025-05-31 02:58:55.529603] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:55.529611] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:58:55.529625] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:55.529646] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:58:55.529659] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:58:55.529679] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:58:55.529701] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:58:55.529711] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:58:55.532356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.542985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.553399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.557337] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:55.563636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.578512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.580671] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:55.590990] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.591486] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=77] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592167] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592191] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592217] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592308] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.592420] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:55.603393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.603998] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:55.604034] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.604067] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.604087] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:55.608260] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-05-31 02:58:55.608624] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=38] ls blacklist refresh finish(cost_time=38497) [2025-05-31 02:58:55.614191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.624736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.635046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.645736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.656417] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.668499] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.679127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.691328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.701553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.707181] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [19591][T1_L0_G1][T1][YB42AC10871E-0006366549CC10D5-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5066) [2025-05-31 02:58:55.708505] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=13, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:55.708554] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=6040, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:55.712338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.722584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.727451] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:378809 disk:48K/11:378809 t1m2:48K/11:378809 t1m2g0:48K/11:378809 |q0: 0 |q1: 0 [2025-05-31 02:58:55.730626] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678335429669000, v:0}, new_max_version={val:1748678335429669000, v:0}, last_min_version={val:1748678335120105001, v:0}, last_max_version={val:1748678335120105001, v:0}, total_time_us=6799, delta_us=300954) [2025-05-31 02:58:55.730680] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678335429669000, v:0}, min_version={val:1748678335429669000, v:0}, max_version={val:1748678335429669000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:55.731198] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:55.731280] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=74] temporary file module use no blocks [2025-05-31 02:58:55.731292] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:55.731307] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:55.731322] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525383913176, flush_io_finished_ret_:0, flush_io_finished_round_:19903, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:55.732891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.732947] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:55.735743] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [19591][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=59] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-05-31 02:58:55.743244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.756029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.761442] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:55.766217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.776610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.781265] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:55.788315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.798567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.805909] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:58:55.805943] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] try gc retain ctx [2025-05-31 02:58:55.805952] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-31 02:58:55.806004] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1748678335722829000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678325630415001, v:0}, keep_alive_scn_:{val:1748678330677627001, v:0}, update_ts_:1748678335805184}) [2025-05-31 02:58:55.806773] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=74] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:58:55.808927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.820225] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4480-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:55.820458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.831684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.841914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.852064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.862285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.872570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.881428] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=30] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:58:55.882785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.893042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.903254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.913530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.923747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.926206] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=41] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:55.928019] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DF9FC-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701206636428833008e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632826557530233913e+03) [2025-05-31 02:58:55.931535] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:55.932038] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:55.934360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.944730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.954876] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.958831] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678335958827, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007314, add_bytes: 0 [2025-05-31 02:58:55.964727] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=98] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:55.965628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.973640] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=28] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:58:55.975906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.981814] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:55.986148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:55.996509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.000642] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:56.006969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.012357] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:56.012407] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=81] PNIO [ratelimit] time: 1748678336012403, bytes: 7131747734, bw: 0.136054 MB/s, add_ts: 1009943, add_bytes: 144081 [2025-05-31 02:58:56.017273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.027493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.031429] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:56.031505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48619, [delay/us]:prepare:12, schedule:170, submit:756, rt:2965, total:3904) [2025-05-31 02:58:56.031525] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:56.031584] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48621, [delay/us]: prepare:12, schedule:170, submit:756, device:2965, total:3904) [2025-05-31 02:58:56.031598] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:56.031625] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4051, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:472, total:487) [2025-05-31 02:58:56.031650] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:56.031691] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4052, [delay/us]: prepare:0, schedule:0, submit:600000000, device:472, total:487) [2025-05-31 02:58:56.031711] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:56.031726] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:56.031735] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:56.031744] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:56.031752] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:56.031761] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=140kB/s, net_out=139kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:56.031782] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399431][2nd=2849]}) [2025-05-31 02:58:56.039190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.049402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.056427] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-05-31 02:58:56.056798] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=19] gc stale ls task succ [2025-05-31 02:58:56.059620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.070393] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:56.072221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.083396] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=13] dump dest keepalive data states, addr: "172.16.135.30:2882", last_write_ts_=1748678335681666, last_read_ts_=1748678335882147, in_black_=0 [2025-05-31 02:58:56.083542] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=21] dump dest keepalive data states, addr: "172.16.135.166:2882", last_write_ts_=1748678335882147, last_read_ts_=1748678335681666, in_black_=0 [2025-05-31 02:58:56.083561] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=11] dump dest keepalive data states, addr: "172.16.135.167:2882", last_write_ts_=1748678335882147, last_read_ts_=1748678335681666, in_black_=0 [2025-05-31 02:58:56.083573] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=6] dump dest keepalive data states, addr: "172.16.135.20:2882", last_write_ts_=1748678335681666, last_read_ts_=1748678335882147, in_black_=0 [2025-05-31 02:58:56.083763] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=5] dump dest keepalive data states, addr: "172.16.135.10:2882", last_write_ts_=1748678335882147, last_read_ts_=1748678335681666, in_black_=0 [2025-05-31 02:58:56.083780] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:679) [81566][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=10] dump dest keepalive data states, addr: "172.16.135.168:2882", last_write_ts_=1748678335681666, last_read_ts_=1748678335882147, in_black_=0 [2025-05-31 02:58:56.084088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.088312] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:56.094314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.107115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.117358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.124077] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2069-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:56.124114] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2069-0-0] [lt=28] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-05-31 02:58:56.128014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.129981] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=90, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:56.130023] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3668, avg_group_log_size=333) [2025-05-31 02:58:56.130044] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3668) [2025-05-31 02:58:56.130058] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:56.132253] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3668) [2025-05-31 02:58:56.137663] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [21284][T1_L0_G1][T1][YB42AC1087A7-00063665517F5515-0-0] [lt=74] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=4535, avg_log_submit_to_slide_time=5244) [2025-05-31 02:58:56.139443] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3668, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:56.140674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.150820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.161308] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=27] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:58:56.161349] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=27] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:58:56.161366] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1044) [2025-05-31 02:58:56.164159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.164968] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:56.174369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.183460] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:56.183767] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A385B-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525385365936, cache_obj->added_lc()=false, cache_obj->get_object_id()=53635, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:56.184713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.194951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.205261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.217730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.227912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.238256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=131][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.250672] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=133][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.260968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.271256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.281587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.291892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.303949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.311042] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3725, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:56.311085] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=531, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:56.311225] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=46, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:56.315577] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.321858] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=40] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:56.321998] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=137] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:56.322083] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F41-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:56.322108] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F41-0-0] [lt=21] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:56.327311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.331686] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=51] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:56.337789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.350952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.360324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.360406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.365933] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:58:56.366128] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=283104, replace_num=15728) [2025-05-31 02:58:56.366159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:56.367989] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.368077] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.368134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.369024] INFO pn_ratelimit (group.c:653) [81063][TimerWK1_IngressService][T0][Y0-0000000000000000-0-0] [lt=27] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:56.371263] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=70] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:58:56.373617] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=44] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:58:56.375800] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=43] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:58:56.375851] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=49] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:58:56.378183] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=65] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:58:56.378324] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=42] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:58:56.378344] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=17] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:58:56.378371] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:58:56.379002] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=626] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:56.379044] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=31] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:58:56.378996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.379056] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:58:56.379078] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:58:56.379098] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:58:56.381293] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:56.381338] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=40] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:56.381350] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:58:56.381385] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:58:56.381399] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:58:56.381408] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:58:56.381421] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:58:56.381437] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDA-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:58:56.386650] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:56.396912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.408060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.418882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.429109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.432443] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678336432420], gts:1748678336432408000, latest_srr:[mts=1748678336432420]}) [2025-05-31 02:58:56.439607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.449915] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.461002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.471909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.482970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.493481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.500916] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:56.504257] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA6F-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:56.505745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.516511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.516620] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:56.527153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.531547] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get number(ret=0, service_type_=0, range=1, base_id=1748678336528202000, start_id=1748678336528202000, end_id=1748678336528202001) [2025-05-31 02:58:56.539341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.547329] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=122] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:56.549691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.562795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.566640] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:56.573122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.583673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.587211] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:56.587486] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 22887424 | cache_store_size= 20807680 | cache_map_size= 2079744 | kv_cnt= 1355 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 52 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 426 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 295 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 473 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 500 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 968 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 8322048 | cache_store_size= 6242304 | cache_map_size= 2079744 | kv_cnt= 76 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 179 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1358 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 27 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-05-31 02:58:56.587519] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] [KVCACHE-HAZARD] hazard version status info: current_version: 1732626 | min_version= 1732626 | total_nodes_count: 0 |  [2025-05-31 02:58:56.588522] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC10870A-0006366715B99DF3-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1748678336587319000, start_id=1748678336587319000, end_id=1748678336587319001) [2025-05-31 02:58:56.590494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591037] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591078] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591105] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591130] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591155] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.591179] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:56.598128] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CE-0-0] [lt=17] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:56.600305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.611154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.621461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.632055] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.645505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.655805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.668218] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81740][T1_TimerWK0_Flush][T1][Y0-0000000000000000-0-0] [lt=49] ====== traversal_flush timer task ====== [2025-05-31 02:58:56.668685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.668576] INFO [MDS] flush (mds_table_mgr.cpp:184) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F23-0-0] [lt=28] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678336528202000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:58:56.668732] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F23-0-0] [lt=149] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:58:56.678961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.689245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.701256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.704921] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=33] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:58:56.707418] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81742][T1_TimerWK1_ReqMemEvict][T1][YB42AC1087A6-0006366543FA5F53-0-0] [lt=67] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=33) [2025-05-31 02:58:56.707709] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678336704416000, start_id=1748678336704416000, end_id=1748678336704416001) [2025-05-31 02:58:56.712053] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:338) [83067][T1_L5_G0][T1][YB42AC1087A7-0006366541CDB88E-0-0] [lt=30][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:11577782}, ref_cnt:1, state:"RUNNING", commit_version:{val:1748678336704416000, v:0}, start_scn:{val:1748678336704416000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1748678366607797, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678336708505000, v:0}, tx_id:{txid:10096478}, scn:{branch:0, seq:27353}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, data_trans_id_:{txid:11577782}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1748678336704416000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1748678336704416000, v:0}}) [2025-05-31 02:58:56.712642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.714039] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [21284][T1_L0_G1][T1][YB42AC10871E-0006366549CC10E0-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3668) [2025-05-31 02:58:56.714357] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-05-31 03:58:56.714531] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [22344][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 03:58:56.715171] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665517F551A-0-0] [lt=67] init point thread id with(&point=0x55c0908e45c0, idx_=1864, point=[thread id=22344, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=22344) [2025-05-31 02:58:56.723144] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=21, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:56.723176] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5991, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:56.723410] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3DC-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525385905580, cache_obj->added_lc()=false, cache_obj->get_object_id()=53636, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:56.724643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.736295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.742798] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678336432408000, v:0}, new_max_version={val:1748678336432408000, v:0}, last_min_version={val:1748678336327001000, v:0}, last_max_version={val:1748678336327001000, v:0}, total_time_us=12698, delta_us=310386) [2025-05-31 02:58:56.742839] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678336432408000, v:0}, min_version={val:1748678336432408000, v:0}, max_version={val:1748678336432408000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:56.748045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.758268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.765852] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=21] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:58:56.765978] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=119] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:56.765993] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:58:56.766005] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:56.768473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.769701] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=20] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:58:56.770580] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=874] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:58:56.770828] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:56.778733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.788886] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.789228] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:56.799140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.809350] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.819554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.822651] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4481-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:56.829822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.841041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.851327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:56.863494] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3B-0-0] [lt=35] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678336858941, keepalive_ts:-1}) [2025-05-31 03:58:56.863553] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3B-0-0] [lt=56] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678336858941, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:56.863862] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.874139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.884452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.888075] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:56.888111] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=25] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:56.888135] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:56.895817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.906137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.916347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.916787] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:199152 disk:52K/12:199152 t1m2:52K/12:199152 t1m2g0:52K/12:199152 |q0: 0 |q1: 0 [2025-05-31 02:58:56.926692] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.926823] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=76] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:56.931622] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1748678336432408000, v:0}, min_version={val:1748678336432408000, v:0}, max_version={val:1748678336432408000, v:0}) [2025-05-31 02:58:56.933345] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678336932023000, start_id=1748678336932023000, end_id=1748678336932023001) [2025-05-31 02:58:56.933388] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:56.936935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.947262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.957526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.964938] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=147] PNIO [ratelimit] time: 1748678336964933, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006106, add_bytes: 0 [2025-05-31 02:58:56.969253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.973923] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:56.980718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:56.984159] INFO [ARCHIVE] do_thread_task_ (ob_archive_timer.cpp:124) [83010][T1_ArcTimer][T1][YB42AC1087A6-000636654299FC95-0-0] [lt=34] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-05-31 02:58:56.989775] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:56.991149] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.002827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.003190] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:57.015232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.017052] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1748678337017049, bytes: 7131950677, bw: 0.192646 MB/s, add_ts: 1004646, add_bytes: 202943 [2025-05-31 02:58:57.017663] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:57.032342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.033956] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA0B-0-0] [lt=43] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.677206087112426758e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.677240859557639396e+03) [2025-05-31 02:58:57.034454] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-05-31 02:58:57.034480] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:57.034762] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81065][TimerWK3_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=101] runTimer to refresh locality_info(ret=0) [2025-05-31 02:58:57.037806] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=17] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:58:57.037983] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=158] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:58:57.038002] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=18] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:58:57.038026] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:58:57.038109] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=82] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:58:57.038133] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=8] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:58:57.038149] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:58:57.038161] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:58:57.039437] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:57.039505] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44698, [delay/us]:prepare:21, schedule:274, submit:153, rt:2345, total:2794) [2025-05-31 02:58:57.039522] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:57.039625] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=86] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44696, [delay/us]: prepare:21, schedule:274, submit:153, device:2345, total:2794) [2025-05-31 02:58:57.039639] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:57.039665] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4063, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:525, total:541) [2025-05-31 02:58:57.039691] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:57.039720] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4063, [delay/us]: prepare:0, schedule:0, submit:600000000, device:525, total:541) [2025-05-31 02:58:57.039738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:57.039751] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:57.040247] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=491] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:57.040259] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:57.040268] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:57.040276] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=201kB/s, net_out=120kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:57.040298] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399432][2nd=29]}) [2025-05-31 02:58:57.047253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.058996] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=31] current monitor number(seq_=-1) [2025-05-31 02:58:57.059090] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=22] gc stale ls task succ [2025-05-31 02:58:57.059248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.069540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.071776] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:57.079733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.090914] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:57.091313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.101750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.114484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:57.118378] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [22494][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:57.118948] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:58:57.131287] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.141527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.151936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.165484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.175097] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:57.179033] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33C3-0-0] [lt=19][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525386361202, cache_obj->added_lc()=false, cache_obj->get_object_id()=53637, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:57.179291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=495][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.191801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=215][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.192827] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A385D-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525386374997, cache_obj->added_lc()=false, cache_obj->get_object_id()=53638, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:57.194058] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:57.195454] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=817882, clean_num=31457) [2025-05-31 02:58:57.204370] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.216621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.217159] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=140] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=90, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:57.217202] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3514, avg_group_log_size=351) [2025-05-31 02:58:57.217272] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3981) [2025-05-31 02:58:57.217316] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:57.225209] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3514) [2025-05-31 02:58:57.225392] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=128, avg_log_submit_to_flush_time=6065, avg_log_submit_to_slide_time=6607) [2025-05-31 02:58:57.225689] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3514, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:57.229464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.240337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.253478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.266657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.277498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.287877] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.298040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.308311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.318654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.331894] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.344664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.353029] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=11] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:58:57.354553] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=48] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:58:57.354677] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=43] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678332169852000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:58:57.356175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.359011] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.359053] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.359112] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.359134] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.366669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.380331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.383034] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:57.395030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.404763] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=104] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:57.407163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.418448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.421789] INFO [STORAGE] get_macro_block_list (ob_sn_tmp_file_manager.cpp:187) [81065][TimerWK3_BlkMgr][T1][Y0-0000000000000000-0-0] [lt=38] get tmp file macro block list(ret=0, ret="OB_SUCCESS", macro_id_list.count()=0) [2025-05-31 02:58:57.427339] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=43] block manager free block(macro_id={[ver=1,mode=0,seq=399420][2nd=155]}, io_fd={first_id:1152921504607246396, second_id:155, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427393] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=51] block manager free block(macro_id={[ver=1,mode=0,seq=399421][2nd=2591]}, io_fd={first_id:1152921504607246397, second_id:2591, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427407] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=13] block manager free block(macro_id={[ver=1,mode=0,seq=399422][2nd=3027]}, io_fd={first_id:1152921504607246398, second_id:3027, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427416] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399403][2nd=2348]}, io_fd={first_id:1152921504607246379, second_id:2348, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427424] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399404][2nd=2925]}, io_fd={first_id:1152921504607246380, second_id:2925, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427432] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399405][2nd=2502]}, io_fd={first_id:1152921504607246381, second_id:2502, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427440] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399406][2nd=2361]}, io_fd={first_id:1152921504607246382, second_id:2361, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427448] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399407][2nd=2643]}, io_fd={first_id:1152921504607246383, second_id:2643, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427456] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399408][2nd=2938]}, io_fd={first_id:1152921504607246384, second_id:2938, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427464] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399409][2nd=2092]}, io_fd={first_id:1152921504607246385, second_id:2092, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427472] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399410][2nd=2374]}, io_fd={first_id:1152921504607246386, second_id:2374, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427480] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399411][2nd=2656]}, io_fd={first_id:1152921504607246387, second_id:2656, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427488] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399412][2nd=2810]}, io_fd={first_id:1152921504607246388, second_id:2810, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427495] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399413][2nd=2669]}, io_fd={first_id:1152921504607246389, second_id:2669, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427503] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399414][2nd=2951]}, io_fd={first_id:1152921504607246390, second_id:2951, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427511] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399415][2nd=2528]}, io_fd={first_id:1152921504607246391, second_id:2528, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427518] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399416][2nd=2682]}, io_fd={first_id:1152921504607246392, second_id:2682, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427526] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399417][2nd=2964]}, io_fd={first_id:1152921504607246393, second_id:2964, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427534] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399418][2nd=2259]}, io_fd={first_id:1152921504607246394, second_id:2259, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427542] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399419][2nd=2541]}, io_fd={first_id:1152921504607246395, second_id:2541, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427549] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399424][2nd=2823]}, io_fd={first_id:1152921504607246400, second_id:2823, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427557] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] block manager free block(macro_id={[ver=1,mode=0,seq=399423][2nd=2400]}, io_fd={first_id:1152921504607246399, second_id:2400, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427565] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399425][2nd=2977]}, io_fd={first_id:1152921504607246401, second_id:2977, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427601] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=33] block manager free block(macro_id={[ver=1,mode=0,seq=399426][2nd=2554]}, io_fd={first_id:1152921504607246402, second_id:2554, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427610] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] block manager free block(macro_id={[ver=1,mode=0,seq=399427][2nd=2836]}, io_fd={first_id:1152921504607246403, second_id:2836, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427618] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399428][2nd=2413]}, io_fd={first_id:1152921504607246404, second_id:2413, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427626] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399429][2nd=2695]}, io_fd={first_id:1152921504607246405, second_id:2695, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427634] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399430][2nd=2567]}, io_fd={first_id:1152921504607246406, second_id:2567, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427642] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399431][2nd=2849]}, io_fd={first_id:1152921504607246407, second_id:2849, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427650] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399432][2nd=29]}, io_fd={first_id:1152921504607246408, second_id:29, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:58:57.427747] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1023) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] finish once mark and sweep(ret=0, alloc_num=42458, mark_info.count()=30, marker_status={total_block_count:3072, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:31, shared_data_block_count:12, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:0, pending_free_count:30, free_count:3013, shared_meta_block_count:14, mark_cost_time:5664, sweep_cost_time:335, start_time:"2025-05-31 03:58:57.421662", last_end_time:"2025-05-31 03:58:57.427661", hold_info:nothing, mark_finished:true}, map_cnt=57) [2025-05-31 02:58:57.430650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.430955] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4565, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:57.431005] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1948, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:57.432106] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=63, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:57.434256] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15] ts response task statistics(alloc_count=486242, free_count=243120) [2025-05-31 02:58:57.434316] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678336933403], gts:1748678336970599000, latest_srr:[mts=1748678337434288]}) [2025-05-31 02:58:57.440948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.451711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.462562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.473083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.485918] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.497056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.505364] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=94] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:57.507796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.518279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.519028] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:57.529023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.530165] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83080][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA70-0-0] [lt=124][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:57.540153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.551286] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.561505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.571927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.582359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.583213] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:57.591356] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A6C-0-0] [lt=81] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:58:57.592638] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.592711] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.592742] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.592769] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.592794] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.592821] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.593333] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:57.593658] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A5651-0-0] [lt=49] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:58:57.593721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.599183] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=62] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:57.603906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.605284] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:57.615409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.626913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.637342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.650000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.651785] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:57.651863] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] get_max_applied_scn(scn={val:1748678337439567000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137251575}, palf_committed_end_scn_:{val:1748678337642109000, v:0}, last_check_scn_:{val:1748678337642109000, v:0}, max_applied_cb_scn_:{val:1748678337439567000, v:0}}) [2025-05-31 02:58:57.652108] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=235] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137251575}, readable_log_tail:{lsn:2137251575}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10101001, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:58:57.660465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.673342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.683650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.694834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=142][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.705665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=618][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.718964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.719060] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:57.723316] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [21284][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=42] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-05-31 02:58:57.729908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.733051] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:57.733092] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-05-31 02:58:57.733103] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:57.733119] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:57.733134] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525385913757, flush_io_finished_ret_:0, flush_io_finished_round_:19905, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:57.742473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.753063] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=54] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:58:57.755066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.765476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.775764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.776546] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=97] LogLoopThread round_cost_time(us)(round_cost_time=21) [2025-05-31 02:58:57.787295] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:58:57.787803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=196][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.798081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.807606] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:57.808369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.818694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.823023] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC10EB-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4102) [2025-05-31 02:58:57.823598] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:57.823766] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=166] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7356, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:57.825297] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4482-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:57.829484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.839641] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.848909] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:1126916 disk:48K/11:1126916 t1m2:48K/11:1126916 t1m2g0:48K/11:1126916 |q0: 0 |q1: 0 [2025-05-31 02:58:57.850376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.851692] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=155] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678337439567001, v:0}, new_max_version={val:1748678337439567001, v:0}, last_min_version={val:1748678337236488001, v:0}, last_max_version={val:1748678337236488001, v:0}, total_time_us=8074, delta_us=412122) [2025-05-31 02:58:57.851746] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678337439567001, v:0}, min_version={val:1748678337439567001, v:0}, max_version={val:1748678337439567001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:57.863055] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.873592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.885059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.896139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.906397] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.916660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.927138] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=74] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:57.928764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.935380] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678337434540], gts:1748678337470558000, latest_srr:[mts=1748678337935349]}) [2025-05-31 02:58:57.942557] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.953799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=598][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.965713] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=68] PNIO [ratelimit] time: 1748678337965709, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000776, add_bytes: 0 [2025-05-31 02:58:57.965815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.976626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.984647] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=12] [APPLY STAT CB APPEND COST TIME](cur_stat_count=51, stat_interval=5000000, avg (count/cost)=161, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:58:57.984685] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=37] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=51, stat_interval=5000000, avg (count/cost)=6860, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:58:57.984694] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=51, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:58:57.984701] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=51, stat_interval=5000000, avg (count/cost)=-1222, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:58:57.984708] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3856-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=51, stat_interval=5000000, avg (count/cost)=5799, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:58:57.986976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:57.989061] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:58:57.997241] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.005691] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:58.006576] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=86] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:58:58.008222] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:58.010166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.017222] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1748678338017218, bytes: 7132279974, bw: 0.313989 MB/s, add_ts: 1000169, add_bytes: 329297 [2025-05-31 02:58:58.020025] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:58.021539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.031763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.042401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.053587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.053963] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:58.054010] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48449, [delay/us]:prepare:14, schedule:1095, submit:751, rt:3068, total:4929) [2025-05-31 02:58:58.054026] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:58.054083] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48451, [delay/us]: prepare:14, schedule:1095, submit:751, device:3068, total:4929) [2025-05-31 02:58:58.054198] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=108] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:58.054226] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4037, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2973, total:2986) [2025-05-31 02:58:58.054253] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:58.054285] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4037, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2973, total:2986) [2025-05-31 02:58:58.054303] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:58.054316] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:58.054325] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:58.054333] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:58.054341] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:58.054349] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=321kB/s, net_out=248kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:58.054369] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399433][2nd=2144]}) [2025-05-31 02:58:58.060094] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2025-05-31 02:58:58.060179] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=18] gc stale ls task succ [2025-05-31 02:58:58.063768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.072327] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=57] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:58.076153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.086408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.091241] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=124] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:58.098224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.108977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.119183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.119233] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:58:58.119386] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [22494][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:58:58.129400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.135464] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206A-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:58:58.135521] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206A-0-0] [lt=47] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=58) [2025-05-31 02:58:58.139692] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.149944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.160237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.164001] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AEE-0-0] [lt=45] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:58.173949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.184272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.189395] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:58:58.194802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.203960] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83107][T1_L0_G0][T1][YB42AC1087A6-000636654C3A385F-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525387386127, cache_obj->added_lc()=false, cache_obj->get_object_id()=53639, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.205552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.209205] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:58.218155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.225479] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=81, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:58.225593] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3980, avg_group_log_size=398) [2025-05-31 02:58:58.225616] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3980) [2025-05-31 02:58:58.225632] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:58.228531] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3980) [2025-05-31 02:58:58.228690] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=127, avg_log_submit_to_flush_time=5989, avg_log_submit_to_slide_time=6710) [2025-05-31 02:58:58.228904] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.229261] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3980, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:58.240830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.248538] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA1D-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701206636428833008e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632826557530233913e+03) [2025-05-31 02:58:58.251138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.263580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.273820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.285864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.292351] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5957-0-0] [lt=28] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678333606640, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:58.292460] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5957-0-0] [lt=106] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678333609641, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:58.292488] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5957-0-0] [lt=22] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678333615187, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678333615192, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:58:58.295512] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE41D9-0-0] [lt=211] get number(ret=0, service_type_=0, range=1, base_id=1748678338294771000, start_id=1748678338294771000, end_id=1748678338294771001) [2025-05-31 02:58:58.296176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.308672] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.319012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.323321] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=49] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:58.323416] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=93] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:58:58.323485] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F42-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:58.323509] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F42-0-0] [lt=19] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:58:58.329271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.331962] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=50] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:58:58.340621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.351157] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.359387] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.359451] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.359483] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.359510] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.361377] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.371792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.382069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.393824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=106] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:58:58.394085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.404431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.410610] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:58.414640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.425148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.433388] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4397, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:58.433429] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=600, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:58.433513] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=103, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:58.435819] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:58.439357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.451157] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=72] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:58:58.455503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.465792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.476166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.477828] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3861-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525387659997, cache_obj->added_lc()=false, cache_obj->get_object_id()=53640, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.479023] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3863-0-0] [lt=127][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:58:58.480556] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3865-0-0] [lt=25][errcode=-4201] hashtable not init [2025-05-31 02:58:58.484048] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3866-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678338483450000, start_id=1748678338483450000, end_id=1748678338483450001) [2025-05-31 02:58:58.486955] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:58.488510] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [22962][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:58.489550] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-05-31 02:58:58.501474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.506953] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-05-31 02:58:58.511968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.522219] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:58.522489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.533437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.543683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.550691] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA71-0-0] [lt=82][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:58.555260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.565673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.578342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.588953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.590647] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.590695] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.590724] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.590749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.590776] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.594442] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.594657] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:58:58.594718] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.594762] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.594785] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.594844] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:58.596138] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=112] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=298832, replace_num=15728) [2025-05-31 02:58:58.596164] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:58:58.599396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.606302] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20CF-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:58.610172] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=39698) [2025-05-31 02:58:58.610313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.611302] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:58.620466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.630786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.633232] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A34-0-0] [lt=133][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525387815402, cache_obj->added_lc()=false, cache_obj->get_object_id()=53641, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:58.641321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.651600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.662215] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.671917] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=37] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:58:58.671965] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=31] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1027) [2025-05-31 02:58:58.672466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.682728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.692939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.703165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.715165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.725517] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E75-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678338724636000, start_id=1748678338724636000, end_id=1748678338724636001) [2025-05-31 02:58:58.726082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.734950] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=59] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:58.735037] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-05-31 02:58:58.735052] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:58.735068] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:58.735083] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525386915378, flush_io_finished_ret_:0, flush_io_finished_round_:19906, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:58.736425] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.746696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.756911] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.767117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.777353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.787534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.797295] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:58:58.797939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.808706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.809522] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:58:58.809622] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=90] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:58:58.809649] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:58:58.812306] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:58.818093] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:58.818225] WDIAG read_multi_row (ob_table_access_helper.h:238) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:58:58.820219] INFO delete_row (ob_table_access_helper.h:339) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678338818244) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678338818244) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:58:58.820768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.828227] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4483-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:58.831102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.836004] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC10F6-0-0] [lt=43] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4546) [2025-05-31 02:58:58.839222] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:58.839330] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=107] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5837, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:58.842190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.852769] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.863325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.865336] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=253] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678338483450001, v:0}, new_max_version={val:1748678338483450001, v:0}, last_min_version={val:1748678338245404001, v:0}, last_max_version={val:1748678338245404001, v:0}, total_time_us=8302, delta_us=381883) [2025-05-31 02:58:58.865378] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678338483450001, v:0}, min_version={val:1748678338483450001, v:0}, max_version={val:1748678338483450001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:58.865402] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:58:58.874488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.885413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.886107] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:58:58.895683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.904058] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81740][T1_TimerWK0_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=44] clear dtl interm result cost(us)(clear_cost=3703, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:58:58.906290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.917096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.927391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.927418] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=91] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:58.933001] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=36] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:58:58.937716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.937759] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:58:58.948006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.958292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.969267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.971478] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=83] PNIO [ratelimit] time: 1748678338971473, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005764, add_bytes: 0 [2025-05-31 02:58:58.980131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.990381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:58.997481] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:58:59.000612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.007630] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:59.012824] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:59.013561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.022445] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=75] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:59.023794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.024763] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678339024743, bytes: 7132442081, bw: 0.153443 MB/s, add_ts: 1007525, add_bytes: 162107 [2025-05-31 02:58:59.034105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.034643] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=33] ====== check clog disk timer task ====== [2025-05-31 02:58:59.034674] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:58:59.036239] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=41] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:137961 disk:48K/11:137961 t1m2:48K/11:137961 t1m2g0:48K/11:137961 |q0: 0 |q1: 0 [2025-05-31 02:58:59.044393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.049735] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9E-0-0] [lt=44] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678339083706, keepalive_ts:-1}) [2025-05-31 02:58:59.049771] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9E-0-0] [lt=34] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678339083706, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:58:59.055191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.061007] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=47] gc stale ls task succ [2025-05-31 02:58:59.065829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.072767] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:58:59.072826] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44223, [delay/us]:prepare:13, schedule:171, submit:504, rt:2749, total:3439) [2025-05-31 02:58:59.072940] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:59.073013] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44218, [delay/us]: prepare:13, schedule:171, submit:504, device:2749, total:3439) [2025-05-31 02:58:59.073029] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:58:59.073071] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4020, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:237, total:250) [2025-05-31 02:58:59.073099] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:58:59.073162] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4020, [delay/us]: prepare:0, schedule:0, submit:600000000, device:237, total:250) [2025-05-31 02:58:59.073182] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:58:59.073198] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:59.073206] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:58:59.073215] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:59.073222] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:58:59.073231] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=150kB/s, net_out=138kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:58:59.073251] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399434][2nd=2426]}) [2025-05-31 02:58:59.073396] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:58:59.078895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=163][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.089204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.092975] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:58:59.099821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.112171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.122939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.135057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.145754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.156448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.168078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.181951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.194723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.197706] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=107] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:58:59.205405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.213758] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3867-0-0] [lt=50][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525388395924, cache_obj->added_lc()=false, cache_obj->get_object_id()=53642, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:59.214462] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:59.216463] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=849339, clean_num=31457) [2025-05-31 02:58:59.216709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.227143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.236755] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=93, this=0x2b09131e5470, extra_info="") [2025-05-31 02:58:59.236918] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=147] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3998, avg_group_log_size=399) [2025-05-31 02:58:59.236953] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3998) [2025-05-31 02:58:59.237052] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:58:59.238155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.242701] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3998) [2025-05-31 02:58:59.242792] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=137, avg_log_submit_to_flush_time=5486, avg_log_submit_to_slide_time=5813) [2025-05-31 02:58:59.242910] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3998, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:58:59.248689] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.259003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.263506] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA2C-0-0] [lt=47] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701358032226562500e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632549326822011153e+03) [2025-05-31 02:58:59.263631] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0D8E-0-0] [lt=56] handle gts request statistics(total_rt=1721, total_cnt=170, avg_rt=1.006432748538011701e+01, avg_cnt=1.700000000000000000e+01) [2025-05-31 02:58:59.269608] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.279986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.293080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.293704] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=39] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:58:59.305085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.315989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.326539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.336824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.347781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.358170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.358680] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.362460] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.362504] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.362531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.367608] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:455) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=31] skip defragment task(ret=0, need_defragment=false) [2025-05-31 02:58:59.367644] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:497) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=33] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=11) [2025-05-31 02:58:59.367657] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:515) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=11] successfully defragment data blocks(rewrite_cnt=0) [2025-05-31 02:58:59.367668] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [81743][T1_TimerWK2_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=349942, total_skip_cnt=6068, not_in_mem_tablet_cnt=516, invalid_attr_tablet_cnt=0) [2025-05-31 02:58:59.368903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.371991] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=28] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:58:59.379091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.389425] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.398148] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=86] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:58:59.402013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.412567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.417128] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:59.422813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.426531] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=31] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-05-31 02:58:59.433151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.437506] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678339437456], gts:1748678339436094000, latest_srr:[mts=1748678339437456]}) [2025-05-31 02:58:59.437567] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=58] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:58:59.437637] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:58:59.442727] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3785, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:58:59.442789] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=874, this=0x2b09049fd870, extra_info="") [2025-05-31 02:58:59.442923] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131effe0, extra_info="") [2025-05-31 02:58:59.446024] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.456726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.466919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.471064] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [22962][T1_L0_G1][T1][YB42AC10871E-0006366549CC10FC-0-0] [lt=39] init point thread id with(&point=0x55c0908ee040, idx_=2482, point=[thread id=22962, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=22962) [2025-05-31 02:58:59.471672] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678339263543001, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678339471631}, version_delta=208126) [2025-05-31 02:58:59.477318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.478798] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3869-0-0] [lt=39][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525388660966, cache_obj->added_lc()=false, cache_obj->get_object_id()=53643, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:58:59.487623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.487675] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:58:59.489727] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [8165][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=17] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-05-31 02:58:59.498023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.507983] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:58:59.508563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.518900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.522659] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=137] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:58:59.530983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.541270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.551569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.564206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.572015] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA72-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:59.574379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.584770] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.590487] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.590538] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.590566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.590590] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.590616] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.591650] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.591952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=94] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:58:59.595232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.600798] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83107][T1_L0_G0][T1][YB42AC1087A8-000636672F3E3F9E-0-0] [lt=63] get number(ret=0, service_type_=0, range=1, base_id=1748678339600269000, start_id=1748678339600269000, end_id=1748678339600269001) [2025-05-31 02:58:59.601311] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:58:59.606499] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:59.609109] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [23361][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:58:59.610075] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-05-31 02:58:59.617814] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:59.620508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.632833] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.643682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=226][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.656896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.671457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:58:59.676961] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [23361][T1_L0_G1][T1][YB42AC1087A7-00063665517F5538-0-0] [lt=28] init point thread id with(&point=0x55c0908f4400, idx_=2881, point=[thread id=23361, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=23361) [2025-05-31 02:58:59.681761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.692141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.702603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.714303] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83107][T1_L0_G0][T1][YB42AC108714-000636671E7D5D70-0-0] [lt=30] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=282, Holding Memory(MB)=134, Tenant Remain Memory(MB)=247, Usable Remain Memory(MB)=148, Last Update Limit Timestamp=1748678339711956, Is Updated=true) [2025-05-31 02:58:59.714554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.727391] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC10871E-000636654378A518-0-0] [lt=53] get number(ret=0, service_type_=0, range=1, base_id=1748678339727380000, start_id=1748678339727380000, end_id=1748678339727380001) [2025-05-31 02:58:59.728280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.730668] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1748678329717011, end_time=1748678339730664, dag_status={}) [2025-05-31 02:58:59.730809] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=136] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.730828] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731047] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=216] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731060] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731074] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731086] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731094] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:58:59.731109] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:58:59.731235] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=123] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:58:59.735408] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:58:59.735446] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-05-31 02:58:59.735457] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:58:59.735472] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:58:59.735529] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=55] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525387917298, flush_io_finished_ret_:0, flush_io_finished_round_:19907, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:58:59.735980] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:58:59.740118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.750958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.761705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.771996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.784087] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.794380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.802566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:58:59.804980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.816787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.818700] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:58:59.827027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.832753] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4484-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:58:59.833740] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1152) [81740][T1_TimerWK0_TabletStatRpt][T1][Y0-0000000000000000-0-0] [lt=49] refresh queuing mode(ret=0, tenant_id=1, stream_cnt=19, update_schema_cnt=19, cost_time=87) [2025-05-31 02:58:59.833771] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1202) [81740][T1_TimerWK0_TabletStatRpt][T1][Y0-0000000000000000-0-0] [lt=27] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1, bucket_cnt:0, tenant_mem_user_limit:1073741824, tenant_mem_user_hold:209457152, cached_node_array:0x2b090499dce0}) [2025-05-31 02:58:59.837973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=269][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.846546] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1101-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4955) [2025-05-31 02:58:59.847601] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=41, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:58:59.847630] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5944, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:58:59.850382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.860591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.865210] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678339663972000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=201227, in_cluster_service=true, cluster_version={val:1748678339460924000, v:0}, min_cluster_version={val:1748678339460924000, v:0}, max_cluster_version={val:1748678339460924000, v:0}, get_cluster_version_err=0, cluster_version_delta=404275, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195242, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=330810441, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:58:59.867285] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=89] BlockGCTimerTask success(ret=0, cost_time_us=18, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:58:59.870846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.877455] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19336736768, fast_delete_log_mode=false) [2025-05-31 02:58:59.882432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.882703] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678339600269000, v:0}, new_max_version={val:1748678339600269000, v:0}, last_min_version={val:1748678339460924000, v:0}, last_max_version={val:1748678339460924000, v:0}, total_time_us=17405, delta_us=282430) [2025-05-31 02:58:59.882777] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678339600269000, v:0}, min_version={val:1748678339600269000, v:0}, max_version={val:1748678339600269000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:58:59.894803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.905644] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:72, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183754797347, recv_hp_rpc_cnt:74878, recv_np_rpc_cnt:877590, recv_lp_rpc_cnt:0, recv_mysql_cnt:47029, recv_task_cnt:2472, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1231737 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9875, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:543801, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1748678339606616}{group_id:10, queue_size:0, recv_req_cnt:103912, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18356, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678338119230}{group_id:2, queue_size:0, recv_req_cnt:150867, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101068, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:107881, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272421, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=683 pcode=0x717:cnt=491 pcode=0x1502:cnt=212 pcode=0x710:cnt=145 pcode=0x523:cnt=99, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:58:59.906572] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=925] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82968438, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:58:59.907308] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=733] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84464067, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:58:59.908006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=689][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.918198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.928430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.929328] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:58:59.939173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.949375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.964608] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.971799] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678339971795, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000322, add_bytes: 0 [2025-05-31 02:58:59.974821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.985460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:58:59.997936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.002763] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:00.008816] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.009304] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=10398720) [2025-05-31 02:59:00.019402] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:00.019640] INFO [COMMON] wash (ob_kvcache_store.cpp:355) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash time detail, (compute_wash_size_time=188, refresh_score_time=41, wash_time=0) [2025-05-31 02:59:00.020165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.025263] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=67] PNIO [ratelimit] time: 1748678340025258, bytes: 7132627775, bw: 0.177000 MB/s, add_ts: 1000515, add_bytes: 185694 [2025-05-31 02:59:00.025350] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=83] PNIO client:0x2b09681fe048_"172.16.135.166:42586"_"172.16.135.168:2882"_210_1748666653311900_1, write_queue=0/0, write=13851/16739106, read=13851/4124815, doing=0, done=13851, write_time=3258885, read_time=21726, process_time=373777 [2025-05-31 02:59:00.025363] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO client:0x2b0933382a98_"172.16.135.166:59348"_"172.16.135.20:2882"_195_1748666366947224_1, write_queue=0/0, write=13765/16831247, read=13764/4185646, doing=0, done=13765, write_time=3105903, read_time=63454, process_time=376070 [2025-05-31 02:59:00.025371] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b09333fc048_"172.16.135.166:51602"_"172.16.135.10:2882"_187_1748666213570986_1, write_queue=0/0, write=34016/29169928, read=34015/38448145, doing=0, done=34016, write_time=7862770, read_time=221484, process_time=986959 [2025-05-31 02:59:00.025379] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b08f8dfe048_"172.16.135.166:45077"_"172.16.135.166:2882"_173_1748658395788060_1, write_queue=0/0, write=100734/65220769, read=100734/24398812, doing=0, done=100734, write_time=12728557, read_time=13012, process_time=2064425 [2025-05-31 02:59:00.025388] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8d6ea98_"172.16.135.166:36999"_"172.16.135.167:2882"_146_1748658389703621_1, write_queue=0/0, write=472088/263585461, read=472088/108416577, doing=0, done=472088, write_time=94890972, read_time=729706, process_time=9819715 [2025-05-31 02:59:00.025396] INFO pn_print_diag_info (group.c:755) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b08f8c86048_"172.16.135.166:49187"_"172.16.135.30:2882"_130_1748658388755113_1, write_queue=0/0, write=484740/285769049, read=484740/138883787, doing=0, done=484740, write_time=98971872, read_time=2290933, process_time=11695501 [2025-05-31 02:59:00.025403] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09681fea98_"172.16.135.91:37414"_211_1748666654745010, write_queue=0/0, write=117681/80646911, read=117681/121517850, doing=0, done=117681, write_time=22065307, read_time=1839404, process_time=8468266 [2025-05-31 02:59:00.025411] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2b09333854a8_"172.16.135.20:35122"_199_1748666369107382, write_queue=0/0, write=122081/89497505, read=122081/126598674, doing=0, done=122081, write_time=23141280, read_time=4141242, process_time=10986198 [2025-05-31 02:59:00.025418] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09333fd4e8_"172.16.135.10:36502"_185_1748666216201478, write_queue=0/0, write=137905/88645185, read=137905/160822365, doing=0, done=137905, write_time=25044214, read_time=8501443, process_time=16256971 [2025-05-31 02:59:00.025425] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b0933386048_"172.16.135.90:49146"_177_1748658396131996, write_queue=0/0, write=678158/242253918, read=678158/442465095, doing=0, done=678158, write_time=118473267, read_time=10773441, process_time=52506907 [2025-05-31 02:59:00.025432] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b08f8dff4e8_"172.16.135.166:45077"_175_1748658395791365, write_queue=0/0, write=100734/24398812, read=100734/65220769, doing=0, done=100734, write_time=7676765, read_time=17019, process_time=6515032 [2025-05-31 02:59:00.025439] INFO pn_print_diag_info (group.c:768) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8c04808_"172.16.135.30:45366"_121_1748658387308557, write_queue=0/0, write=701513/261486415, read=701513/465378527, doing=0, done=701513, write_time=127994932, read_time=8109061, process_time=52241369 [2025-05-31 02:59:00.025445] INFO pn_print_diag_info (group.c:775) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:1128810/689756506, client_queue_time=1972670430, cnt=6, server_send:1946491/972872493, server_queue_time=2785195693, cnt=6 [2025-05-31 02:59:00.027008] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:00.028118] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:59:00.028284] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=28] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:59:00.030565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.044591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.056758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.061138] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=29] current monitor number(seq_=-1) [2025-05-31 02:59:00.061307] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=77] gc stale ls task succ [2025-05-31 02:59:00.069913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.070362] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:215941 disk:52K/12:215941 t1m2:52K/12:215941 t1m2g0:52K/12:215941 |q0: 0 |q1: 0 [2025-05-31 02:59:00.074186] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:00.076435] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:00.081440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.085191] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:00.085248] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56640, [delay/us]:prepare:31, schedule:229, submit:137, rt:3300, total:3699) [2025-05-31 02:59:00.085265] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:00.085323] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56646, [delay/us]: prepare:31, schedule:229, submit:137, device:3300, total:3699) [2025-05-31 02:59:00.085336] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:00.085363] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4046, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:344, total:359) [2025-05-31 02:59:00.085387] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:00.085415] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4046, [delay/us]: prepare:0, schedule:0, submit:600000000, device:344, total:359) [2025-05-31 02:59:00.085433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:00.085446] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:00.085455] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:00.085463] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:00.085472] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:00.085480] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=196kB/s, net_out=124kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:00.085500] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399435][2nd=2862]}) [2025-05-31 02:59:00.095253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.095472] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:00.099496] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81743][T1_TimerWK2_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=13] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:00.104369] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB946DB-0-0] [lt=11][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:1474001454, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678340129564, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1384221, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678340115916, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678340104364, enqueue_timestamp_:0, request_arrival_time_:1748678340104364, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:00.104478] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB946DB-0-0] [lt=68][errcode=-5150] deliver rpc request fail(&req=0x2b099d616098, ret=-5150) [2025-05-31 02:59:00.108163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.114660] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=53][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aecad0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038512} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:7133} write_epoch=1, prev=this=0x2b095e513f98 trans_version={val:1748676712277139000, v:0} scn={val:1748676712277139000, v:0} tx_id={txid:10036630} prev=0x2b095e513de0 next=(nil) modify_count=7 acc_checksum=3200386038 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678340076994000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:24830} write_epoch=3, snapshot_version={val:1748678340109493000, v:0}, *this={this=0x2b09b3158800 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e513f98 latest_compact_node=0x2b09802ce3d0 max_trans_version={val:1748678279867715000, v:0} max_trans_id=10038441 max_elr_trans_version={val:1748678279867715000, v:0} max_elr_trans_id=10038441 latest_compact_ts=1748675024402105 last_compact_cnt=5 total_trans_node_cnt=8 max_modify_scn={val:1748676712277139000, v:0} min_modify_scn={val:1748674842707220000, v:0}}) [2025-05-31 02:59:00.114722] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=60][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aecad0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038512} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:7133} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678370009657, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678370009657, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678340109493000, v:0}, tx_id:{txid:10038512}, scn:{branch:0, seq:7130}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038512}, tx_desc:{this:0x2b094e84b6c0, tx_id:{txid:10038512}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678340114095, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678340140108, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:7133}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:00.114816] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=93][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038512}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7133}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:00.114960] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=140][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":4},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038512} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:00.114999] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=36][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.115016] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09b6c0c820, local_datums:0x2b09b6c0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09b6c0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09b6c0c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f1, hex: 4D5C4AE769360600, int: 1748678340140109}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f9, hex: 4D5C4AE769360600, int: 1748678340140109}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c501, hex: 0194600700000000, int: 123769857, num_digit0: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c509, hex: 00F0FD0800000000, int: 150859776}]}) [2025-05-31 02:59:00.115210] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:45) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=113] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=60666, insert_dup_release_count=60666, col_map_alloc_count=0, col_map_release_count=0) [2025-05-31 02:59:00.115244] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=19][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.119364] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11266}, to_seq_no={branch:0, seq:7131}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038512} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.119470] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=48] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-05-31 02:59:00.119536] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:148) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E0-0-0] [lt=64] transaction rpc statistics(total_rt=4618, total_process=22, avg_rt=200) [2025-05-31 02:59:00.119771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.130081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.132007] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC10871E-00063665464899E1-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1748678340130757000, start_id=1748678340130757000, end_id=1748678340130757001) [2025-05-31 02:59:00.132277] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83061][T1_L0_G0][T1][YB42AC10871E-00063665464899E1-0-0] [lt=99] transaction rpc statistics(total_rt=25311, total_process=85, avg_rt=294) [2025-05-31 02:59:00.134393] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO client:0x2b09681fca98_"172.16.135.166:42582"_"172.16.135.168:2882"_206_1748666651298651_1, write_queue=0/0, write=13610/16407711, read=13610/5220374, doing=0, done=13610, write_time=3042911, read_time=101784, process_time=436946 [2025-05-31 02:59:00.134464] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=67] PNIO client:0x2b09333834e8_"172.16.135.166:59352"_"172.16.135.20:2882"_196_1748666368752601_1, write_queue=0/0, write=14094/17118249, read=14094/5008131, doing=0, done=14094, write_time=3016923, read_time=161034, process_time=475519 [2025-05-31 02:59:00.134476] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO client:0x2b09333fca98_"172.16.135.166:51614"_"172.16.135.10:2882"_188_1748666215568366_1, write_queue=0/0, write=33969/28946310, read=33966/37377301, doing=0, done=33969, write_time=7488765, read_time=173145, process_time=942029 [2025-05-31 02:59:00.134485] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8dfea98_"172.16.135.166:37013"_"172.16.135.167:2882"_174_1748658395788183_1, write_queue=0/0, write=472833/266160275, read=472833/108756591, doing=0, done=472833, write_time=95548949, read_time=645047, process_time=9840529 [2025-05-31 02:59:00.134493] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b08f8d6e048_"172.16.135.166:49199"_"172.16.135.30:2882"_138_1748658389703238_1, write_queue=0/0, write=483169/283273090, read=483168/139603159, doing=0, done=483169, write_time=95950968, read_time=2651999, process_time=11830438 [2025-05-31 02:59:00.134501] INFO pn_print_diag_info (group.c:755) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8c86a98_"172.16.135.166:45053"_"172.16.135.166:2882"_131_1748658388755503_1, write_queue=0/0, write=101411/65196175, read=101411/24537182, doing=0, done=101411, write_time=12481590, read_time=12012, process_time=2120686 [2025-05-31 02:59:00.134507] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09681ff4a8_"172.16.135.91:37418"_212_1748666654931158, write_queue=0/0, write=117353/77912579, read=117353/121434326, doing=0, done=117353, write_time=21331343, read_time=4776104, process_time=11457497 [2025-05-31 02:59:00.134515] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b0933384a98_"172.16.135.20:35114"_198_1748666369094799, write_queue=0/0, write=122066/86833349, read=122066/124807346, doing=0, done=122066, write_time=22785301, read_time=1740711, process_time=8490351 [2025-05-31 02:59:00.134521] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09333ff468_"172.16.135.10:36510"_192_1748666216307410, write_queue=0/0, write=137888/95593880, read=137888/161070883, doing=0, done=137888, write_time=25590202, read_time=4411836, process_time=12064839 [2025-05-31 02:59:00.134527] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8ccd468_"172.16.135.30:45398"_145_1748658389366474, write_queue=0/0, write=701629/257624333, read=701629/464004551, doing=0, done=701629, write_time=125708916, read_time=7842126, process_time=51483341 [2025-05-31 02:59:00.134533] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8ccca58_"172.16.135.90:49126"_140_1748658389037157, write_queue=0/0, write=678356/240856823, read=678356/441985327, doing=0, done=678356, write_time=118467063, read_time=11245753, process_time=52109573 [2025-05-31 02:59:00.134539] INFO pn_print_diag_info (group.c:768) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8ccc048_"172.16.135.166:45053"_133_1748658388758769, write_queue=0/0, write=101411/24537182, read=101411/65196175, doing=0, done=101411, write_time=7343779, read_time=16915, process_time=6571792 [2025-05-31 02:59:00.134550] INFO pn_print_diag_info (group.c:775) [81429][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO client_send:1128796/689453792, client_queue_time=1970622515, cnt=6, server_send:1947105/972090021, server_queue_time=2766204550, cnt=6 [2025-05-31 02:59:00.138158] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206B-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:00.138191] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206B-0-0] [lt=24] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=35) [2025-05-31 02:59:00.140449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.142409] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83079][T1_L0_G0][T1][YB42AC10871E-00063665464899E2-0-0] [lt=42][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:00.142494] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83079][T1_L0_G0][T1][YB42AC10871E-00063665464899E2-0-0] [lt=79][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.142514] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83079][T1_L0_G0][T1][YB42AC10871E-00063665464899E2-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b097d20c820, local_datums:0x2b097d20c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b097d20c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b097d20c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c4f3, hex: 30D64AE769360600, int: 1748678340171312}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c4fb, hex: 30D64AE769360600, int: 1748678340171312}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b097d20c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:00.142715] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83079][T1_L0_G0][T1][YB42AC10871E-00063665464899E2-0-0] [lt=89][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.144526] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E2-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7278}, to_seq_no={branch:0, seq:3891}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038513} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.150927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.155106] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E3-0-0] [lt=91] get number(ret=0, service_type_=0, range=1, base_id=1748678340154494000, start_id=1748678340154494000, end_id=1748678340154494001) [2025-05-31 02:59:00.163975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.170940] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=9][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aecd50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038514} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:4971} write_epoch=1, prev=this=0x2b0980347320 trans_version={val:1748678279934553000, v:0} scn={val:1748678279934553000, v:0} tx_id={txid:10038443} prev=0x2b0988adf6a0 next=(nil) modify_count=115 acc_checksum=2212027381 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678340076994000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:14957} write_epoch=3, snapshot_version={val:1748678340168461000, v:0}, *this={this=0x2b097ccc9760 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0980347320 latest_compact_node=0x2b0980344938 max_trans_version={val:1748678279934553000, v:0} max_trans_id=10038443 max_elr_trans_version={val:1748678279934553000, v:0} max_elr_trans_id=10038443 latest_compact_ts=1748678219648929 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678279934553000, v:0} min_modify_scn={val:1748674842846408000, v:0}}) [2025-05-31 02:59:00.171006] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aecd50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038514} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:4971} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678370066827, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678370066827, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678340168461000, v:0}, tx_id:{txid:10038514}, scn:{branch:0, seq:4968}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038514}, tx_desc:{this:0x2b094e8bcbc0, tx_id:{txid:10038514}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678340169601, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678340199300, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:4971}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:00.171148] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=140][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038514}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4971}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:00.171168] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":4},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038514} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:00.171200] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.171216] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09b6c0c820, local_datums:0x2b09b6c0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09b6c0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09b6c0c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f1, hex: 85434BE769360600, int: 1748678340199301}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f9, hex: 85434BE769360600, int: 1748678340199301}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c501, hex: 7010662700000000, int: 661000304}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c509, hex: 7010662700000000, int: 661000304}]}) [2025-05-31 02:59:00.171330] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.172829] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899E4-0-0] [lt=56] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7415}, to_seq_no={branch:0, seq:4969}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038514} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.174246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.189330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.201130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.207178] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=118] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:00.212323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.219422] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aed378 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038516} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4968} write_epoch=1, prev=this=0x2b095e514c90 trans_version={val:1748676712371193000, v:0} scn={val:1748676712371193000, v:0} tx_id={txid:10036634} prev=0x2b095e514ad8 next=(nil) modify_count=7 acc_checksum=4123576988 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678340076994000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:18514} write_epoch=3, snapshot_version={val:1748678340216775000, v:0}, *this={this=0x2b09802c6d88 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e514c90 latest_compact_node=0x2b09802ce4a0 max_trans_version={val:1748678279974494000, v:0} max_trans_id=10038445 max_elr_trans_version={val:1748678279974494000, v:0} max_elr_trans_id=10038445 latest_compact_ts=1748675024503164 last_compact_cnt=5 total_trans_node_cnt=8 max_modify_scn={val:1748676712371193000, v:0} min_modify_scn={val:1748674843182584000, v:0}}) [2025-05-31 02:59:00.219489] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=64][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aed378 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038516} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4968} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678370115310, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678370115310, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678340216775000, v:0}, tx_id:{txid:10038516}, scn:{branch:0, seq:4965}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038516}, tx_desc:{this:0x2b094e9a56e0, tx_id:{txid:10038516}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678340218070, commit_ts:-1, finish_ts:-1, timeout_us:29999987, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678340247719, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:4968}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:00.219584] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038516}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4968}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:00.219603] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":4},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038516} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:00.219633] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.219648] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09b6c0c820, local_datums:0x2b09b6c0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09b6c0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09b6c0c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f2, hex: A8004CE769360600, int: 1748678340247720}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4fa, hex: A8004CE769360600, int: 1748678340247720}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c502, hex: 0040D10900000000, int: 164708352}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c50a, hex: 0040D10900000000, int: 164708352}]}) [2025-05-31 02:59:00.219905] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=129][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.220427] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:00.221447] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899E8-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7911}, to_seq_no={branch:0, seq:4966}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038516} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.222523] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.228818] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A386B-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525389410988, cache_obj->added_lc()=false, cache_obj->get_object_id()=53644, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:00.232924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.242450] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=15][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aed5f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038517} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:3692} write_epoch=1, prev=this=0x2b095e515030 trans_version={val:1748676712396784000, v:0} scn={val:1748676712396784000, v:0} tx_id={txid:10036635} prev=0x2b097cd10eb8 next=(nil) modify_count=3 acc_checksum=3427276312 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678340076994000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:17483} write_epoch=3, snapshot_version={val:1748678340240107000, v:0}, *this={this=0x2b09b315a578 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b095e515030 latest_compact_node=(nil) max_trans_version={val:1748678279994283000, v:0} max_trans_id=10038446 max_elr_trans_version={val:1748678279994283000, v:0} max_elr_trans_id=10038446 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748676712396784000, v:0} min_modify_scn={val:1748674843232767000, v:0}}) [2025-05-31 02:59:00.242995] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=542][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aed5f8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038517} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:3692} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678370140298, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678370140298, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678340240107000, v:0}, tx_id:{txid:10038517}, scn:{branch:0, seq:3689}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038517}, tx_desc:{this:0x2b094e8964a0, tx_id:{txid:10038517}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678340240107, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678340272394, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:3692}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:00.243100] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=103][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038517}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3692}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:00.243119] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":4},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038517} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:00.243201] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.243243] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=14][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09b6c0c820, local_datums:0x2b09b6c0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09b6c0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09b6c0c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f1, hex: 0B614CE769360600, int: 1748678340272395}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f9, hex: 0B614CE769360600, int: 1748678340272395}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c501, hex: 0020190000000000, int: 1646592}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c509, hex: 0020190000000000, int: 1646592}]}) [2025-05-31 02:59:00.243357] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=61][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.244702] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899EA-0-0] [lt=121] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6229}, to_seq_no={branch:0, seq:3690}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038517} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.245569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:00.245990] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [23567][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:00.247582] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:59:00.254958] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899EB-0-0] [lt=206] get number(ret=0, service_type_=0, range=1, base_id=1748678340253889000, start_id=1748678340253889000, end_id=1748678340253889001) [2025-05-31 02:59:00.255104] INFO [LIB] stat (utility.h:1211) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899EB-0-0] [lt=35] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=94, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:00.255135] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899EB-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10387, avg_group_log_size=546) [2025-05-31 02:59:00.255154] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899EB-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=9950) [2025-05-31 02:59:00.255168] INFO [LIB] stat (utility.h:1211) [83090][T1_L0_G0][T1][YB42AC10871E-00063665464899EB-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:00.257982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.258181] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=10387) [2025-05-31 02:59:00.258263] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=129, avg_log_submit_to_flush_time=4375, avg_log_submit_to_slide_time=5345) [2025-05-31 02:59:00.258320] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10387, hit_cnt=20, read_cnt=20, hit rate=1.000000000000000000e+00) [2025-05-31 03:59:00.259375] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [23570][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:00.259550] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-05-31 02:59:00.268329] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=93][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:00.268528] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899ED-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1748678340268070000, start_id=1748678340268070000, end_id=1748678340268070001) [2025-05-31 02:59:00.271451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.279753] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=30] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.677357482910156250e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.676957770654617889e+03) [2025-05-31 02:59:00.279947] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=59][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b097cd45658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038519} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:4185} write_epoch=1, prev=this=0x2b098030e170 trans_version={val:1748676712445819000, v:0} scn={val:1748676712445819000, v:0} tx_id={txid:10036637} prev=0x2b098030e0a0 next=(nil) modify_count=5 acc_checksum=52101520 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678340076994000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:16353} write_epoch=0, snapshot_version={val:1748678340275165000, v:0}, *this={this=0x2b097ccc9fb0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098030e170 latest_compact_node=0x2b098030e170 max_trans_version={val:1748678280026255000, v:0} max_trans_id=10038448 max_elr_trans_version={val:1748678280026255000, v:0} max_elr_trans_id=10038448 latest_compact_ts=1748676712452327 last_compact_cnt=5 total_trans_node_cnt=6 max_modify_scn={val:1748676712445819000, v:0} min_modify_scn={val:1748674843351980000, v:0}}) [2025-05-31 02:59:00.279999] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=50][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b097cd45658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10038519} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:90} seq_no={branch:0, seq:4185} write_epoch=1, ctx={this:0x2b091cc551a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678370177625, tablet_id:{id:309}, table_iter:0x2b091cc536d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678370177625, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678340275165000, v:0}, tx_id:{txid:10038519}, scn:{branch:0, seq:4182}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10038519}, tx_desc:{this:0x2b098b2ff820, tx_id:{txid:10038519}, state:1, addr:"172.16.135.30:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678340279544, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678340306519, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cb39d0, mem_ctx:0x2b0904cb4f50, tx_scn:{branch:0, seq:4185}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:00.280166] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=165][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10038519}, data:0x2b091cc520c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4185}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:00.280188] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.30", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":4},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10038519} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:00.280219] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.280233] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098ee0c820, local_datums:0x2b098ee0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098ee0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b098ee0c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c4f1, hex: E7EA4CE769360600, int: 1748678340307687}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c4f9, hex: E7EA4CE769360600, int: 1748678340307687}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c501, hex: 0050D30000000000, int: 13848576}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098ee0c509, hex: 0050D30000000000, int: 13848576}]}) [2025-05-31 02:59:00.280377] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83097][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=99][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.282187] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83079][T1_L0_G0][T1][YB42AC10871E-00063665464899EE-0-0] [lt=112] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9632}, to_seq_no={branch:0, seq:4183}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038519} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.284428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:00.294056] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [23570][T1_L0_G1][T1][YB42AC10871E-0006366549CC110F-0-0] [lt=69] init point thread id with(&point=0x55c0908f7840, idx_=3090, point=[thread id=23570, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=23570) [2025-05-31 02:59:00.296077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.306429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.307125] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83103][T1_L0_G0][T1][YB42AC10871E-00063665464899F1-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1748678340306247000, start_id=1748678340306247000, end_id=1748678340306247001) [2025-05-31 02:59:00.316774] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.322020] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899F2-0-0] [lt=11][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:00.322066] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899F2-0-0] [lt=41][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:00.322087] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899F2-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09b6c0c820, local_datums:0x2b09b6c0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09b6c0c4c4, hex: 3137322E31362E3133352E3330, cstr: 172.16.135.30}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4d9, hex: 0400000000000000, int: 4}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09b6c0c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f0, hex: FA8A4DE769360600, int: 1748678340348666}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c4f8, hex: FA8A4DE769360600, int: 1748678340348666}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09b6c0c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:00.322252] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10871E-00063665464899F2-0-0] [lt=67][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:00.323982] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10871E-00063665464899F2-0-0] [lt=36] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9521}, to_seq_no={branch:0, seq:5445}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10038521} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:00.331173] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83089][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA77-0-0] [lt=88][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525389513344, cache_obj->added_lc()=false, cache_obj->get_object_id()=53645, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:00.331485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=256][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.332094] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [81743][T1_TimerWK2_CompactionRefre][T1][YB42AC1087A6-00063665430A5ED1-0-0] [lt=75] InfoPoolResizeTask(cost_ts=8) [2025-05-31 02:59:00.332221] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=31] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:00.332400] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=178] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:00.332506] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F43-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:00.332587] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F43-0-0] [lt=76] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:00.334925] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=42] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:00.341783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.349744] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO client:0x2b09681e5468_"172.16.135.166:42578"_"172.16.135.168:2882"_203_1748666651183223_1, write_queue=0/0, write=13524/16416304, read=13524/5479991, doing=0, done=13524, write_time=3040908, read_time=169768, process_time=507654 [2025-05-31 02:59:00.349777] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO client:0x2b0933382048_"172.16.135.166:59346"_"172.16.135.20:2882"_194_1748666366893400_1, write_queue=0/0, write=13886/16644955, read=13886/5202056, doing=0, done=13886, write_time=3045901, read_time=151406, process_time=455863 [2025-05-31 02:59:00.349787] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client:0x2b09333874a8_"172.16.135.166:51600"_"172.16.135.10:2882"_186_1748666213446445_1, write_queue=0/0, write=33979/28708790, read=33978/38650108, doing=0, done=33979, write_time=7398752, read_time=441369, process_time=1204067 [2025-05-31 02:59:00.349796] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8dfd4a8_"172.16.135.166:49211"_"172.16.135.30:2882"_172_1748658395787650_1, write_queue=0/0, write=485818/284998674, read=485815/139078807, doing=0, done=485818, write_time=97209808, read_time=3097505, process_time=12290342 [2025-05-31 02:59:00.349832] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO client:0x2b08f8dfa048_"172.16.135.166:45071"_"172.16.135.166:2882"_150_1748658390718795_1, write_queue=0/0, write=100531/64816935, read=100531/24344910, doing=0, done=100531, write_time=12332590, read_time=11297, process_time=2052379 [2025-05-31 02:59:00.349842] INFO pn_print_diag_info (group.c:755) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8c874e8_"172.16.135.166:36989"_"172.16.135.167:2882"_132_1748658388756028_1, write_queue=0/0, write=471553/263932531, read=471553/106788988, doing=0, done=471553, write_time=94260171, read_time=382127, process_time=9444988 [2025-05-31 02:59:00.349943] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=98] PNIO server:0x2b09681fd4e8_"172.16.135.91:37390"_208_1748666651758052, write_queue=0/0, write=117559/77125627, read=117559/121198300, doing=0, done=117559, write_time=21449334, read_time=921563, process_time=7529806 [2025-05-31 02:59:00.349953] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO server:0x2b09681e4048_"172.16.135.20:35130"_201_1748666369372744, write_queue=0/0, write=121763/79352082, read=121763/124122098, doing=0, done=121763, write_time=22446370, read_time=1767323, process_time=8466126 [2025-05-31 02:59:00.349961] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09333fe048_"172.16.135.10:36506"_190_1748666216213783, write_queue=0/0, write=137716/92235826, read=137716/160716858, doing=0, done=137716, write_time=25194164, read_time=4594403, process_time=12226718 [2025-05-31 02:59:00.349968] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b08f8dfca98_"172.16.135.30:45422"_155_1748658392265486, write_queue=0/0, write=700692/258990301, read=700693/464657027, doing=0, done=700693, write_time=124898914, read_time=9617991, process_time=52486338 [2025-05-31 02:59:00.349975] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b08f8dfb4a8_"172.16.135.90:49140"_152_1748658391079697, write_queue=0/0, write=678530/241818425, read=678530/441609739, doing=0, done=678530, write_time=118103078, read_time=5973198, process_time=46486942 [2025-05-31 02:59:00.349981] INFO pn_print_diag_info (group.c:768) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b08f8dfaa98_"172.16.135.166:45071"_151_1748658390719742, write_queue=0/0, write=100531/24344910, read=100531/64816935, doing=0, done=100531, write_time=7254751, read_time=16824, process_time=6390384 [2025-05-31 02:59:00.349987] INFO pn_print_diag_info (group.c:775) [81431][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:1128788/687759515, client_queue_time=1947337770, cnt=6, server_send:1945190/965461906, server_queue_time=2747662005, cnt=6 [2025-05-31 02:59:00.352272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.359092] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.359684] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.359724] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.359750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.363187] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=33] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:00.364713] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=94] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:00.364899] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=44] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678332169852000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:00.365385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.368766] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=121] ObArchiveSender is running(thread_index=0) [2025-05-31 02:59:00.376154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.387039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.397789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.397801] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=1994, gc_interval=10000000) [2025-05-31 02:59:00.397942] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=109] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:59:00.397966] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=55) [2025-05-31 02:59:00.397980] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:59:00.397992] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:59:00.398177] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=120] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:59:00.398198] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=20] execute_gc cost time(ret=0, time_us=24) [2025-05-31 02:59:00.398209] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] ObSafeDestroyHandler start process [2025-05-31 02:59:00.398217] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:59:00.398238] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:59:00.407408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:00.408568] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.418885] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.420977] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:00.429214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.439096] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15] ts response task statistics(alloc_count=486314, free_count=243156) [2025-05-31 02:59:00.439138] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=741, get_gts_with_stc_cnt=42, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=42, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:00.439156] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:00.439207] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678339938359], gts:1748678339974290000, latest_srr:[mts=1748678340439171]}) [2025-05-31 02:59:00.440272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.458047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.464218] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT WRITE LOG COST TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=3543, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:00.464274] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=1003, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:00.466384] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=63, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:00.468257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.481526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.491922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.502210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.509809] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:59:00.514826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.526107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.527890] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=106] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:00.535343] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D5132-0-0] [lt=55][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525389717512, cache_obj->added_lc()=false, cache_obj->get_object_id()=53646, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:00.536572] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=27] ====== checkpoint timer task ====== [2025-05-31 02:59:00.537341] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=42] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:00.537425] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=84] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:00.538121] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=59] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:00.538211] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:00.538271] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:00.538286] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1748678333710865001, v:0}) [2025-05-31 02:59:00.538294] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:00.538301] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:00.538316] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:00.538336] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:00.538349] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:00.538369] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:00.538391] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:00.538400] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:00.539104] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81065][TimerWK3_AutoLSLocRpc][T0][YB42AC1087A6-0006366545EA56CF-0-0] [lt=13] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:59:00.539308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.549553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.560461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.570213] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1}, delta=101304, log_service_ts=1748678340468579, min_tx_service_ts=4611686018427387, timestamp={val:1748678340468579000, v:0}) [2025-05-31 02:59:00.570272] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=61, Other_Error_Cnt=0, Submit_Succ_Cnt=39, last_scn="{val:1748678335722829001, v:0}", last_lsn={lsn:2137244572}, last_gts={val:1748678340468579000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:59:00.571298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.584230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.589177] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A6D-0-0] [lt=12] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:00.590339] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.590387] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.590417] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.591585] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.591625] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.591652] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.591925] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:00.594297] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83095][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA73-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:00.596083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.606737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.607613] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:00.615919] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D0-0-0] [lt=70] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:00.618175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.621442] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=56] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:00.630741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.641064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.653166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.663406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.675207] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1748678340674232000, start_id=1748678340674232000, end_id=1748678340674232001) [2025-05-31 02:59:00.675807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.687220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.698027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.708495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.720743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.730999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.738232] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:00.738273] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-05-31 02:59:00.738285] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:00.738300] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:00.738316] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525388917744, flush_io_finished_ret_:0, flush_io_finished_round_:19908, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:00.741487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.752005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.762460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.772815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.783119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.794712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.805243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.808864] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=314560, replace_num=15728) [2025-05-31 02:59:00.808973] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=106] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:00.809192] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=27] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137272232}, readable_log_tail:{lsn:2137272232}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10080344, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:00.809293] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=100] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:59:00.809327] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=11] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:59:00.815530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.819630] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b09681fc048_"172.16.135.166:42580"_"172.16.135.168:2882"_205_1748666651231145_1, write_queue=0/0, write=13455/16426452, read=13455/3880843, doing=0, done=13455, write_time=2984908, read_time=35913, process_time=366151 [2025-05-31 02:59:00.819659] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO client:0x2b0933384048_"172.16.135.166:59354"_"172.16.135.20:2882"_197_1748666369056438_1, write_queue=0/0, write=13900/16859860, read=13899/4640929, doing=0, done=13900, write_time=3025911, read_time=99038, process_time=396792 [2025-05-31 02:59:00.819669] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client:0x2b0933386a58_"172.16.135.166:51596"_"172.16.135.10:2882"_183_1748666213382799_1, write_queue=0/0, write=33898/28867353, read=33898/38178926, doing=0, done=33898, write_time=7254791, read_time=201788, process_time=965569 [2025-05-31 02:59:00.819678] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x2b08f8dfc048_"172.16.135.166:49209"_"172.16.135.30:2882"_154_1748658391730300_1, write_queue=0/0, write=484201/283148039, read=484201/138577601, doing=0, done=484201, write_time=96192973, read_time=2118903, process_time=11159373 [2025-05-31 02:59:00.819686] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b08f8dd7468_"172.16.135.166:37003"_"172.16.135.167:2882"_149_1748658390717917_1, write_queue=0/0, write=472474/264048650, read=472474/105766308, doing=0, done=472474, write_time=93731981, read_time=386900, process_time=9354422 [2025-05-31 02:59:00.819694] INFO pn_print_diag_info (group.c:755) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client:0x2b08f8d6f4e8_"172.16.135.166:45067"_"172.16.135.166:2882"_147_1748658389703764_1, write_queue=0/0, write=101506/65159314, read=101506/24558010, doing=0, done=101506, write_time=12315654, read_time=11531, process_time=2064104 [2025-05-31 02:59:00.819701] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09681f2048_"172.16.135.91:37420"_213_1748666655144238, write_queue=0/0, write=117662/91517546, read=117662/122891017, doing=0, done=117662, write_time=21766293, read_time=1623563, process_time=8269632 [2025-05-31 02:59:00.819710] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x2b09681e4a58_"172.16.135.20:35132"_202_1748666369377989, write_queue=0/0, write=122097/77853883, read=122097/124602401, doing=0, done=122097, write_time=21934342, read_time=1265720, process_time=7905054 [2025-05-31 02:59:00.819717] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b09333fea58_"172.16.135.10:36508"_191_1748666216217679, write_queue=0/0, write=137817/95699643, read=137817/159442259, doing=0, done=137817, write_time=25263237, read_time=2069534, process_time=9705632 [2025-05-31 02:59:00.819724] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8dd6a58_"172.16.135.90:49138"_128_1748658390052200, write_queue=0/0, write=678114/241223859, read=678114/443501513, doing=0, done=678114, write_time=118106484, read_time=10239500, process_time=50654478 [2025-05-31 02:59:00.819730] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x2b08f8dd6048_"172.16.135.166:45067"_148_1748658389705519, write_queue=0/0, write=101506/24558010, read=101506/65159314, doing=0, done=101506, write_time=7299760, read_time=17699, process_time=6686928 [2025-05-31 02:59:00.819737] INFO pn_print_diag_info (group.c:768) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x2b08f8c05218_"172.16.135.30:45386"_127_1748658388337082, write_queue=0/0, write=701664/258732418, read=701665/468048966, doing=0, done=701665, write_time=125363913, read_time=10304630, process_time=53459770 [2025-05-31 02:59:00.819743] INFO pn_print_diag_info (group.c:775) [81433][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO client_send:1128793/686746617, client_queue_time=1950663834, cnt=6, server_send:1947329/976900018, server_queue_time=2754567702, cnt=6 [2025-05-31 02:59:00.822067] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=98] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:00.825850] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.837254] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:00.837293] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=30] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:59:00.837317] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:00.837349] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:687) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=6) [2025-05-31 02:59:00.837365] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:593) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=12] TxData Memory Statistic : (Tenant Total Memory(MB)=1024, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=6, Freeze TxData Trigger Memory(MB)=20, Total TxDataTable Hold Memory(MB)=12, Total TxDataTable Memory Limit(MB)=204) [2025-05-31 02:59:00.837365] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4485-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:00.837836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.848182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.860781] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:00.860829] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=45] try gc retain ctx [2025-05-31 02:59:00.860840] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-05-31 02:59:00.860959] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] finish update min start scn(max_decided_scn={val:1748678340674232000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678330677627001, v:0}, keep_alive_scn_:{val:1748678335722829001, v:0}, update_ts_:1748678340860643}) [2025-05-31 02:59:00.861664] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=73] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:00.862018] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.864236] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=68] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=11156) [2025-05-31 02:59:00.865220] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1116-0-0] [lt=184] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=20, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:00.865267] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1116-0-0] [lt=45] [PALF STAT LOG LIFE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=4894, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:00.872233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.881724] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678340468579001, v:0}, new_max_version={val:1748678340468579001, v:0}, last_min_version={val:1748678340438291000, v:0}, last_max_version={val:1748678340438291000, v:0}, total_time_us=8423, delta_us=413141) [2025-05-31 02:59:00.882603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.894484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.894791] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3C-0-0] [lt=12] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678340893022, keepalive_ts:-1}) [2025-05-31 02:59:00.894841] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3C-0-0] [lt=48] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678340893022, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 03:59:00.895026] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [23779][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:00.895291] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-05-31 02:59:00.900195] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032279, max_lsn:2137272934, max_scn:{val:1748678340875026000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137272934}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095329894}}, last_submit_lsn:{lsn:2137272345}, last_submit_end_lsn:{lsn:2137272934}, last_submit_log_id:4032279, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137272345}, max_flushed_end_lsn:{lsn:2137272934}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137272934}, last_slide_log_id:4032279, last_slide_scn:{val:1748678340875026000, v:0}, last_slide_lsn:{lsn:2137272345}, last_slide_end_lsn:{lsn:2137272934}, last_slide_log_pid:11, last_slide_log_accum_checksum:2967608510, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032280, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137272934}, readable_log_tail:{lsn:2137272934}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10079642, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:59:00.900550] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=346] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678340878044, last_flushed_end_lsn:{lsn:2137272934}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678340881096, last_flushed_end_lsn:{lsn:2137272934}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678340885701, last_flushed_end_lsn:{lsn:2137272934}}]) [2025-05-31 02:59:00.900576] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032280, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678340878044, last_flushed_end_lsn:{lsn:2137272934}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678340881096, last_flushed_end_lsn:{lsn:2137272934}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678340885701, last_flushed_end_lsn:{lsn:2137272934}}]) [2025-05-31 02:59:00.905462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.919390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.931245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.931322] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=49] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:00.940088] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] get number(ret=0, service_type_=0, range=1, base_id=1748678340939932000, start_id=1748678340939932000, end_id=1748678340939932001) [2025-05-31 02:59:00.940486] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:00.940532] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678340439592], gts:1748678340475448000, latest_srr:[mts=1748678340940462]}) [2025-05-31 02:59:00.941458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.951705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.961932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.972389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.980727] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678340980722, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008927, add_bytes: 0 [2025-05-31 02:59:00.980778] INFO pn_print_diag_info (group.c:775) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=49] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2025-05-31 02:59:00.982552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:00.994729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.005144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.010439] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=55] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:59:01.013598] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:01.015805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.022562] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:01.030227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.030551] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:01.031977] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO [ratelimit] time: 1748678341031973, bytes: 7132989133, bw: 0.342319 MB/s, add_ts: 1006715, add_bytes: 361358 [2025-05-31 02:59:01.036833] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=34] ====== check clog disk timer task ====== [2025-05-31 02:59:01.036930] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=88] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:01.038229] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=51] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:59:01.042526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.055148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.061913] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=43] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:96K/21:299151 disk:96K/21:299151 t1m2:96K/21:299151 t1m2g0:96K/21:299151 |q0: 0 |q1: 0 [2025-05-31 02:59:01.061972] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=43] current monitor number(seq_=-1) [2025-05-31 02:59:01.062045] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=19] gc stale ls task succ [2025-05-31 02:59:01.065470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.074366] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:01.077243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.087949] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678340674232001, v:0}, min_version={val:1748678340674232001, v:0}, max_version={val:1748678340674232001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:01.088131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.093993] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:01.094113] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4710, iops:20, norm_iops:5, bw:93384, [delay/us]:prepare:16, schedule:317, submit:247, rt:2630, total:3211) [2025-05-31 02:59:01.094134] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:01.094191] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4710, avg_iops:20, avg_bw:93379, [delay/us]: prepare:16, schedule:317, submit:247, device:2630, total:3211) [2025-05-31 02:59:01.094205] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:01.094231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4059, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3610, total:3624) [2025-05-31 02:59:01.094256] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:01.094284] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4059, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3610, total:3624) [2025-05-31 02:59:01.094302] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:01.094316] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:01.094324] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:01.094333] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:01.094341] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:01.094350] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=345kB/s, net_out=148kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:01.094369] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399436][2nd=2157]}) [2025-05-31 02:59:01.095647] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:01.098362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.108584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.118900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.129064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.139309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.149474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.160039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.163268] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=29] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:01.163319] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=39] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:01.163337] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=954) [2025-05-31 02:59:01.172860] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.183510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.204253] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF0-0-0] [lt=47] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:01.213833] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:01.215536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.215733] INFO [WR] exec_sql_with_retry (ob_wr_collector.cpp:1822) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25] the query timeout is too big!!!(ret=0, old_worker_timeout_ts=9223372036854775807, max_timeout=60000000, common::ObTimeUtility::current_time()=1748678341215732) [2025-05-31 02:59:01.223485] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:01.224826] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=880796, clean_num=31457) [2025-05-31 02:59:01.225737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.227660] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=22] === Run print tenant memory usage task === [2025-05-31 02:59:01.227964] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=276] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 120,934,400 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 649,641,984 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,341,225,526 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 52,809,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 895,873,024 max_mem_memstore_can_get_now= 264,046,576 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:59:01.228144] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,071,053,824 total_hold= 3,321,888,768 used= 1,576,685,568 freelists_hold= 1,494,368,256 total_maps= 2,017 total_unmaps= 806 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 120,934,400 virtual_memory_used= 4,342,378,496 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,000,341,504 count= 477 pushes= 14,523,940 pops= 14,523,463 maps= 1,822 unmaps= 762 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,942 pops= 22,938 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:59:01.228174] INFO print (ob_malloc_time_monitor.cpp:37) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 6683, delta_count= 2, avg_cost_time= 3341 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:59:01.236049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.240419] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A386D-0-0] [lt=104][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525390422589, cache_obj->added_lc()=false, cache_obj->get_object_id()=53647, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:01.246265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.246310] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:59:01.251090] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [17207][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=77] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:59:01.256474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.263072] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=106] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=80, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:01.263115] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=5837, avg_group_log_size=416) [2025-05-31 02:59:01.263136] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=6274) [2025-05-31 02:59:01.263151] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:01.264624] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5837) [2025-05-31 02:59:01.266607] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC111A-0-0] [lt=117] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=4662, avg_log_submit_to_slide_time=5203) [2025-05-31 02:59:01.266976] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5837, hit_cnt=14, read_cnt=14, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:01.267041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.277302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.277539] INFO [SQL] check_session_leak (ob_sql_session_mgr.cpp:645) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=60] get current session count(used_session_count=3, hold_session_count=3, session_leak_count_threshold=100) [2025-05-31 02:59:01.280198] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=111] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:01.287494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.297710] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.308002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.318452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.328713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.339723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.351225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.360382] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.360431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.360488] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.360560] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.364534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.377580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.392089] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=1218] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:01.395124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.402582] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=77] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:01.404653] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=58] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:01.406890] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=43] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:01.406933] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=41] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:01.408751] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:01.408949] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=53] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:01.409029] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=76] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:01.409062] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=20] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:01.409762] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=695] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:01.409867] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=34] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:01.409889] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=19] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:01.409998] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=103] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:01.410025] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=25] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:01.410617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.412978] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:01.413032] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=50] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:01.413046] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:01.413083] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:01.413124] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=39] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:01.413138] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=13] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:01.413150] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:01.413167] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDB-0-0] [lt=16] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:01.414290] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:01.422669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.430510] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:01.433400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.441018] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:01.444307] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.455590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.467594] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.470203] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3200, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:01.470464] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=248] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=442, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:01.470550] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:01.478363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.478352] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA4D-0-0] [lt=37] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701509380340576172e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632272219902613415e+03) [2025-05-31 02:59:01.483989] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [23567][T1_L0_G19][T1][YB42AC10871E-00063665511850D8-0-0] [lt=19] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:01.484097] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [23567][T1_L0_G19][T1][YB42AC10871E-00063665511850D8-0-0] [lt=99] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:01.490815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.504583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.511119] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:59:01.515552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.525974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.531691] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=161] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:01.537188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.547436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.560113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.572003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.583757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.590616] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.590740] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.590799] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.591400] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.591536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.591574] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.591959] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:01.598169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.601198] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:01.601236] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:01.601276] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:01.601298] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:01.616255] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=98] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:01.616414] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:164) [81744][T1_TimerWK3_MediumLoop][T1][YB42AC1087A6-00063665431A60C1-0-0] [lt=22] MediumCheckTask(cost_ts=9) [2025-05-31 02:59:01.616718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.626757] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA74-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:01.627653] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] start to print blacklist info [2025-05-31 02:59:01.628161] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=114] ls blacklist refresh finish(cost_time=55980) [2025-05-31 02:59:01.628229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=307][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.631613] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:01.638950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=161][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.650634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.661118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.670082] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=19] ====== traversal_flush timer task ====== [2025-05-31 02:59:01.670674] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60C2-0-0] [lt=153] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678341473655000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678350364653000, latest_log_ts:{val:1748678333710865001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:01.671409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.671447] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60C2-0-0] [lt=765] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:01.682177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.693175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.704003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.714226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.724039] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:01.726106] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81744][T1_TimerWK3_ReqMemEvict][T1][YB42AC1087A6-00063665431A60C3-0-0] [lt=29] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=43) [2025-05-31 02:59:01.726331] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678341726262000, start_id=1748678341726262000, end_id=1748678341726262001) [2025-05-31 02:59:01.726582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.727799] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=57032066, this={dio_aligned_buf:{buf_write_offset:4006, buf_padding_size:90, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:390144, truncate_used_ts:132226}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=57028608, buf_len=548, total_write_size=50601, total_write_size_after_dio=532480, ob_pwrite_used_ts=478874, count=118, each_pwrite_cost=4058) [2025-05-31 02:59:01.733218] INFO [COMMON] loop_running_dag_net_list (ob_tenant_dag_scheduler.cpp:3894) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] loop running dag net list(ret=0, running_dag_net_list_size=0, slow_dag_net_cnt=0, finished_dag_net_list_size=0) [2025-05-31 02:59:01.733278] INFO [COMMON] loop_finished_dag_net_list (ob_tenant_dag_scheduler.cpp:3917) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=57] loop finsihed dag net list(ret=0, finished_dag_net_list_size=0) [2025-05-31 02:59:01.736695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.738567] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:01.738624] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=48] temporary file module use no blocks [2025-05-31 02:59:01.738638] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:01.738654] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:01.738668] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525389920530, flush_io_finished_ret_:0, flush_io_finished_round_:19909, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:01.747016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.758163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.761127] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC10870A-000636671698E3E0-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525390943297, cache_obj->added_lc()=false, cache_obj->get_object_id()=53648, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:01.768985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.770962] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=44] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:01.770996] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:01.771010] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:01.771020] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:01.774467] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:01.775442] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=970] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:01.781220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.791436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.801677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.814250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.817096] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:01.825346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.836063] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:01.836792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.842316] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4486-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:01.849060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.859636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.870561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.874120] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1121-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4094) [2025-05-31 02:59:01.882556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.883502] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137276326}], scn_array=[cnt:1, {val:1748678341726262001, v:0}], curr_size=122, accum_size=2681, time_cost=12312) [2025-05-31 02:59:01.884381] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:01.884428] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6803, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:01.894805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.894876] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:01.895766] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678341473655001, v:0}, new_max_version={val:1748678341473655001, v:0}, last_min_version={val:1748678341473655000, v:0}, last_max_version={val:1748678341473655000, v:0}, total_time_us=19156, delta_us=422107) [2025-05-31 02:59:01.897753] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [22962][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=141] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-05-31 02:59:01.905148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.910894] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:01.916475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.926733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.934825] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=50] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:01.937178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.948030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.958294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.969745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.974952] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=32] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:01.979932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:01.986108] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1748678341986102, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005380, add_bytes: 0 [2025-05-31 02:59:01.990154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.000471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.010763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.011577] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:59:02.018552] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:02.021702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.032273] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=67] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:02.032351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.037716] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=817] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:02.042250] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=89] PNIO [ratelimit] time: 1748678342042244, bytes: 7133222290, bw: 0.220095 MB/s, add_ts: 1010271, add_bytes: 233157 [2025-05-31 02:59:02.042946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.053146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.062390] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=74] gc stale ls task succ [2025-05-31 02:59:02.063399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.073930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.080110] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:02.084575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.085308] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137277037}], scn_array=[cnt:1, {val:1748678341941636000, v:0}], curr_size=122, accum_size=3392, time_cost=11601) [2025-05-31 02:59:02.089455] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=44] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:530050 disk:52K/12:530050 t1m2:52K/12:530050 t1m2g0:52K/12:530050 |q0: 0 |q1: 0 [2025-05-31 02:59:02.095700] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=362] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678341726262000, v:0}, min_version={val:1748678341726262000, v:0}, max_version={val:1748678341726262000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:02.096154] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:02.096270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.107086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.109638] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:02.109711] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48395, [delay/us]:prepare:13, schedule:596, submit:444, rt:4179, total:5234) [2025-05-31 02:59:02.109730] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:02.109788] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48397, [delay/us]: prepare:13, schedule:596, submit:444, device:4179, total:5234) [2025-05-31 02:59:02.109802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:02.109828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4033, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:455, total:469) [2025-05-31 02:59:02.109909] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:02.109949] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4032, [delay/us]: prepare:0, schedule:0, submit:600000000, device:455, total:469) [2025-05-31 02:59:02.109969] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:02.109984] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:02.109993] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:02.110001] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:02.110009] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:02.110018] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=216kB/s, net_out=213kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:02.110039] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399437][2nd=2439]}) [2025-05-31 02:59:02.118162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.131412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.141678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.151937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.155955] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206C-0-0] [lt=82] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:02.155999] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206C-0-0] [lt=33] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=47) [2025-05-31 02:59:02.164994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.175830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.190414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.198460] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:338) [83107][T1_L0_G0][T1][YB42AC10870A-000636671538EF47-0-0] [lt=43][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:11577841}, ref_cnt:1, state:"RUNNING", commit_version:{val:1748678342189973000, v:0}, start_scn:{val:1748678342189973000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1748678372095283, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678342189973000, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, data_trans_id_:{txid:11577841}, data_sql_sequence_:{branch:0, seq:5}, read_latest_:false, read_uncommitted_:false, scn_:{val:1748678342189973000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1748678342189973000, v:0}}) [2025-05-31 02:59:02.219791] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:02.224441] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33C7-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525391406610, cache_obj->added_lc()=false, cache_obj->get_object_id()=53649, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:02.229589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.238343] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:02.240149] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.249291] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83082][T1_L0_G0][T1][YB42AC1087A6-000636654C3A386F-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525391431458, cache_obj->added_lc()=false, cache_obj->get_object_id()=53650, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:02.250479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.260737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.271057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.282360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.283848] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:02.283967] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5903, avg_group_log_size=454) [2025-05-31 02:59:02.283990] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5436) [2025-05-31 02:59:02.284004] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:02.289702] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5903) [2025-05-31 02:59:02.291226] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1127-0-0] [lt=39] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=510, avg_log_gen_to_submit_time=622, avg_log_submit_to_flush_time=7238, avg_log_submit_to_slide_time=7512) [2025-05-31 02:59:02.291634] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5903, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:02.293041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.301920] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE41F1-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1748678342301404000, start_id=1748678342301404000, end_id=1748678342301404001) [2025-05-31 02:59:02.304712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.318231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.328430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.334137] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=42] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:02.334173] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=35] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:02.336700] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F44-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:02.336748] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F44-0-0] [lt=43] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:02.338622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.348897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.350393] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=37] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:02.358138] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.359284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.359472] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.359599] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.359698] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.369480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.372171] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=12] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:02.379742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.390925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.402321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.413174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.420003] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:02.425315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.436208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.439292] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:02.445510] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678342445481], gts:1748678342444253000, latest_srr:[mts=1748678342445481]}) [2025-05-31 02:59:02.445550] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:02.447964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.458219] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.468731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.479305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.481308] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5931, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:02.481362] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1623, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:02.481428] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=95] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=97, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:02.484304] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA58-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.701509380340576172e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.632272219902613415e+03) [2025-05-31 02:59:02.488372] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:02.491493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.497733] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3871-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525391679902, cache_obj->added_lc()=false, cache_obj->get_object_id()=53651, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:02.500653] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3873-0-0] [lt=72][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:02.501998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.503009] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3875-0-0] [lt=56][errcode=-4201] hashtable not init [2025-05-31 02:59:02.503652] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1748678342502560000, start_id=1748678342502560000, end_id=1748678342502560001) [2025-05-31 02:59:02.512452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.514865] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=10398720) [2025-05-31 02:59:02.523075] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.533982] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:02.535521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.546252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.557231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.569119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.579419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.589702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.589973] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.590029] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.590063] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.591291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.591708] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.591747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.591914] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:02.593913] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81065][TimerWK3_ServerTracerTim][T0][YB42AC1087A6-0006366545EA56D0-0-0] [lt=66] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:02.599923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.610387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=184][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.621997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.622331] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:02.624435] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D1-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:02.632364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.641754] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:02.644284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.651198] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA75-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:02.655247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.665472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.676137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.686672] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.689351] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] get_max_applied_scn(scn={val:1748678342502560000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137281656}, palf_committed_end_scn_:{val:1748678342684433000, v:0}, last_check_scn_:{val:1748678342684433000, v:0}, max_applied_cb_scn_:{val:1748678342502560000, v:0}}) [2025-05-31 02:59:02.689413] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137281656}, readable_log_tail:{lsn:2137281656}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10070920, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:02.696962] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.709384] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.720280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.732111] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.741346] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=55] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:02.741390] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-05-31 02:59:02.741401] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:02.741416] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:02.741432] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525390920909, flush_io_finished_ret_:0, flush_io_finished_round_:19910, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:02.742335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.755749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.761024] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [23567][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AE-0-0] [lt=43] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:02.761158] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [23567][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AE-0-0] [lt=123] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:02.765956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.776190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.786919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.797170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.802888] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC108714-000636671EBE41F4-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1748678342801948000, start_id=1748678342801948000, end_id=1748678342801948001) [2025-05-31 02:59:02.808395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.814033] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [83084][T1_L0_G0][T1][YB42AC10871E-00063665464899F8-0-0] [lt=37] need check schema(new_schema_version=1748674258961328, cached_tenant_schema_version=1748674258961328, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=DELETE FROM __all_ls_meta_table WHERE tenant_id = ? AND svr_ip = ? AND svr_port = ?) [2025-05-31 02:59:02.818694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.822979] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-05-31 02:59:02.823465] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:02.828968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.841235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.842375] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:02.845080] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=33] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:02.845116] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:02.845621] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] update_check_range(min_check_version=1983, max_check_version=1984, global_version=1985) [2025-05-31 02:59:02.848327] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4487-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:02.855726] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-05-31 02:59:02.855811] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=74] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52809300, tenant_id=1) [2025-05-31 02:59:02.855893] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:02.856041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.866364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.869161] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 153 276 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 51 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 18 6 3 0 1 20 2 1 0 500 0 88 252 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 2 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 23562 [2025-05-31 02:59:02.869212] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=25] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 895,873,024 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 455,450,624 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 106,954,752 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 12,603,392 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:02.869592] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 455,450,624 used= 344,455,664 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 20,398,080 used= 20,177,472 count= 32 avg_used= 630,546 block_cnt= 32 chunk_cnt= 20 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,127,040 used= 6,258,688 count= 106 avg_used= 59,044 block_cnt= 106 chunk_cnt= 29 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,850,816 used= 2,408,160 count= 87 avg_used= 27,680 block_cnt= 87 chunk_cnt= 26 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 966,656 used= 474,832 count= 59 avg_used= 8,048 block_cnt= 59 chunk_cnt= 22 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 458,240 used= 441,512 count= 11 avg_used= 40,137 block_cnt= 11 chunk_cnt= 9 mod=CommSysVarFac [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 397,824 used= 381,696 count= 84 avg_used= 4,544 block_cnt= 84 chunk_cnt= 25 mod=[T]ObTraceEvent [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 99,600 used= 25,872 count= 366 avg_used= 70 block_cnt= 99 chunk_cnt= 27 mod=Coro [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 54,432 used= 42,336 count= 63 avg_used= 672 block_cnt= 58 chunk_cnt= 24 mod=[T]ObWarningBuf [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 46,720 used= 32,704 count= 73 avg_used= 448 block_cnt= 43 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 44,848 used= 4,776 count= 199 avg_used= 24 block_cnt= 76 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,512 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 3 mod=SqlSession [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=SqlSessiVarMap [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PlanVaIdx [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=SqlSessionSbloc [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,504 used= 2,304 count= 6 avg_used= 384 block_cnt= 5 chunk_cnt= 5 mod=TableLockNode [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,496 used= 1,696 count= 4 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=SequenceIdMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,888 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=SequenceMap [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=PackStateMap [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,424 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 2 mod=SqlSessiQuerSql [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=ContextsMap [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 344,455,664 used= 336,411,275 count= 32,606 avg_used= 10,317 mod=SUMMARY [2025-05-31 02:59:02.869651] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=48] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:59:02.869669] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:59:02.869690] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,574,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,511,808 used= 1,476,096 count= 186 avg_used= 7,936 block_cnt= 186 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,360,000 used= 1,240,000 count= 625 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,574,720 used= 35,241,449 count= 3,357 avg_used= 10,497 mod=SUMMARY [2025-05-31 02:59:02.869708] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:59:02.869721] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 106,954,752 used= 104,251,392 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 104,251,392 used= 104,057,472 count= 202 avg_used= 515,136 block_cnt= 202 chunk_cnt= 51 mod=CoStack [MEMORY] hold= 104,251,392 used= 104,057,472 count= 202 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:59:02.869742] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:59:02.869758] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:59:02.869771] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:02.869784] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:02.869796] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:59:02.869894] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 12,603,392 used= 8,994,048 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,803,392 used= 5,666,304 count= 714 avg_used= 7,936 block_cnt= 714 chunk_cnt= 4 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 8,994,048 used= 8,827,968 count= 718 avg_used= 12,295 mod=SUMMARY [2025-05-31 02:59:02.869919] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:02.869956] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 649,641,984 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:02.870322] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 180,792,064 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorid()=1, group_id_=1, token=0) [2025-05-31 02:59:02.904149] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [18113][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=142] worker thread exit(tenant_->id()=1, workers_.get_size()=5) [2025-05-31 02:59:02.908171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.918666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.932088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.939274] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=97] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:02.943009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.944202] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=28] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:02.954007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.964401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.976653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.986887] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.987132] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678342987127, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001025, add_bytes: 0 [2025-05-31 02:59:02.997125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:02.998591] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=60, stat_interval=5000000, avg (count/cost)=156, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:02.998635] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=60, stat_interval=5000000, avg (count/cost)=6530, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:02.998647] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=60, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:02.998654] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=60, stat_interval=5000000, avg (count/cost)=-1046, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:02.998661] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=60, stat_interval=5000000, avg (count/cost)=5639, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:03.007522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.015459] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:172) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=524288, config_mem_limit_=32212254, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=12478464, evict_high_mem_level=16106127, evict_low_mem_level=0) [2025-05-31 02:59:03.015497] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:03.018375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.022677] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81063][TimerWK1_DiskUseReport][T1][Y0-0000000000000000-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1748678343021784000, start_id=1748678343021784000, end_id=1748678343021784001) [2025-05-31 02:59:03.027097] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=330288, replace_num=15728) [2025-05-31 02:59:03.027131] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:03.029085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.031572] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=7] get number(ret=0, service_type_=0, range=1, base_id=1748678343029503000, start_id=1748678343029503000, end_id=1748678343029503001) [2025-05-31 02:59:03.031850] INFO [SHARE] delete_tenant_all (ob_disk_usage_table_operator.cpp:229) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=27] delete successful (ret=0, tenant_id=500) [2025-05-31 02:59:03.032001] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:916) [81063][TimerWK1_DiskUseReport][T1][Y0-0000000000000000-0-0] [lt=24] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:03.033793] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [81063][TimerWK1_DiskUseReport][T1][Y0-0000000000000000-0-0] [lt=14] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-05-31 02:59:03.036273] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=18][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09803490c8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577852} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2b098030d388 trans_version={val:1748676659761255000, v:0} scn={val:1748676659761255000, v:0} tx_id={txid:11519521} prev=0x2b097cd0ecf8 next=(nil) modify_count=5 acc_checksum=3738645070 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678342993076000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7} write_epoch=0, snapshot_version={val:1748678343029503001, v:0}, *this={this=0x2b097cccb518 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098030d388 latest_compact_node=0x2b098030d388 max_trans_version={val:1748678282918563001, v:0} max_trans_id=11577201 max_elr_trans_version={val:1748678282918563001, v:0} max_elr_trans_id=11577201 latest_compact_ts=1748676659765552 last_compact_cnt=5 total_trans_node_cnt=6 max_modify_scn={val:1748676659761255000, v:0} min_modify_scn={val:1748674856094996001, v:0}}) [2025-05-31 02:59:03.036324] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=49][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09803490c8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577852} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2b08ee8cc5c0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678372935921, tablet_id:{id:309}, table_iter:0x2b08ee8caaf8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678372935921, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678343029503001, v:0}, tx_id:{txid:11577852}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11577852}, tx_desc:{this:0x2b0991b93380, tx_id:{txid:11577852}, state:1, addr:"172.16.135.166:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1748678343029503, active_ts:1748678343029503, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678343058159, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1748678343029503001, v:0}, ref:2}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:03.036414] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=89][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11577852}, data:0x2b08ee8c94c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:03.036432] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.166", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11577852} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:03.036461] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.036474] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa61e6f0, local_datums:0x2b09aa61e6f0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e57c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa61e584, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e592, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e59a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09aa61e5a2, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5b2, hex: FF7276E769360600, int: 1748678343029503}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5ba, hex: FF7276E769360600, int: 1748678343029503}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5c2, hex: AB4FD70300000000, int: 64442283}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5ca, hex: 0020F20400000000, int: 82976768}]}) [2025-05-31 02:59:03.036574] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.036631] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A565F-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577852} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.040471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.044067] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:03.046278] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=78] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:03.046320] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=131] ====== check clog disk timer task ====== [2025-05-31 02:59:03.046390] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=65] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:03.047073] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1748678343047069, bytes: 7133391062, bw: 0.160181 MB/s, add_ts: 1004825, add_bytes: 168772 [2025-05-31 02:59:03.050808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.051197] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1, file_type=0, data_size=64442283) [2025-05-31 02:59:03.053176] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5662-0-0] [lt=29][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:03.053201] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5662-0-0] [lt=19][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.053215] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5662-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa61e690, local_datums:0x2b09aa61e690}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e51c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa61e524, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e532, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e53a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09aa61e542, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e554, hex: 2DCA76E769360600, int: 1748678343051821}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e55c, hex: 2DCA76E769360600, int: 1748678343051821}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e564, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e56c, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:03.053369] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5662-0-0] [lt=62][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.053428] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5662-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577853} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.061085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.062666] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2025-05-31 02:59:03.062745] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=32] gc stale ls task succ [2025-05-31 02:59:03.063301] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=71] insert successful (ret=0, tenant_id=1, file_type=8, data_size=0) [2025-05-31 02:59:03.065205] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=28][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b09803495c0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577854} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2b097cd43f78 trans_version={val:1748678282938875000, v:0} scn={val:1748678282938875000, v:0} tx_id={txid:11577203} prev=0x2b097cd43dc0 next=(nil) modify_count=115 acc_checksum=3037155881 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678342993076000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1748678343062910000, v:0}, *this={this=0x2b097cccc158 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b097cd43f78 latest_compact_node=0x2b0980344d20 max_trans_version={val:1748678282938875000, v:0} max_trans_id=11577203 max_elr_trans_version={val:1748678282938875000, v:0} max_elr_trans_id=11577203 latest_compact_ts=1748678222845551 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678282938875000, v:0} min_modify_scn={val:1748674856182603002, v:0}}) [2025-05-31 02:59:03.065256] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=48][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b09803495c0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577854} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2b08ee8cc5c0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678372964795, tablet_id:{id:309}, table_iter:0x2b08ee8caaf8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678372964795, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678343062910000, v:0}, tx_id:{txid:11577854}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11577854}, tx_desc:{this:0x2b0991b850e0, tx_id:{txid:11577854}, state:1, addr:"172.16.135.166:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1748678343062910, active_ts:1748678343062910, commit_ts:-1, finish_ts:-1, timeout_us:29999985, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678343090551, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1748678343062910000, v:0}, ref:2}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:03.065348] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=90][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11577854}, data:0x2b08ee8c94c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:03.065422] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=71][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.166", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11577854} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:03.065452] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.065465] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa61e720, local_datums:0x2b09aa61e720}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5ac, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa61e5b4, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5c2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5ca, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09aa61e5d2, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5e2, hex: 7EF576E769360600, int: 1748678343062910}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5ea, hex: 7EF576E769360600, int: 1748678343062910}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5f2, hex: 165E666300000000, int: 1667653142}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e5fa, hex: 165E666300000000, int: 1667653142}]}) [2025-05-31 02:59:03.065567] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.065623] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5665-0-0] [lt=10] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577854} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.070335] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9F-0-0] [lt=29] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678343103082, keepalive_ts:-1}) [2025-05-31 02:59:03.070442] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85B9F-0-0] [lt=105] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678343103082, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:03.071418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.072668] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1, file_type=5, data_size=1667653142) [2025-05-31 02:59:03.079623] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=29] insert successful (ret=0, tenant_id=500, file_type=5, data_size=1677721600) [2025-05-31 02:59:03.082259] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=29][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0980349be8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577856} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2b098030d458 trans_version={val:1748676659786183001, v:0} scn={val:1748676659786183001, v:0} tx_id={txid:11519525} prev=0x2b097cd0f9f0 next=(nil) modify_count=5 acc_checksum=3909977700 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678342993076000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2244} write_epoch=0, snapshot_version={val:1748678343078989000, v:0}, *this={this=0x2b097cccd730 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098030d458 latest_compact_node=0x2b098030d458 max_trans_version={val:1748678282959699000, v:0} max_trans_id=11577205 max_elr_trans_version={val:1748678282959699000, v:0} max_elr_trans_id=11577205 latest_compact_ts=1748676659793080 last_compact_cnt=5 total_trans_node_cnt=6 max_modify_scn={val:1748676659786183001, v:0} min_modify_scn={val:1748674856267486002, v:0}}) [2025-05-31 02:59:03.082368] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=107][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0980349be8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577856} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2b08ee8cc5c0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678372981849, tablet_id:{id:309}, table_iter:0x2b08ee8caaf8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678372981849, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678343078989000, v:0}, tx_id:{txid:11577856}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11577856}, tx_desc:{this:0x2b0991b97440, tx_id:{txid:11577856}, state:1, addr:"172.16.135.166:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1748678343078989, active_ts:1748678343078989, commit_ts:-1, finish_ts:-1, timeout_us:29999919, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678343111021, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1748678343078989000, v:0}, ref:2}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:03.082460] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=91][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11577856}, data:0x2b08ee8c94c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:03.082478] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.166", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11577856} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:03.082507] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.082520] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa6986f0, local_datums:0x2b09aa6986f0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa69857c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa698584, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa698592, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa69859a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09aa6985a2, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985b3, hex: 4D3477E769360600, int: 1748678343078989}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985bb, hex: 4D3477E769360600, int: 1748678343078989}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985c3, hex: 0040C50500000000, int: 96813056}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985cb, hex: 0040C50500000000, int: 96813056}]}) [2025-05-31 02:59:03.082626] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.082680] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566B-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577856} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.083462] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=80] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:03.084037] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.090951] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=30] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:80K/18:468876 disk:80K/18:468876 t1m2:80K/18:468876 t1m2g0:80K/18:468876 |q0: 0 |q1: 0 [2025-05-31 02:59:03.092907] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1, file_type=7, data_size=96813056) [2025-05-31 02:59:03.095087] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=129][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0980349e68 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577858} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2b097ccdb670 trans_version={val:1748675156950562001, v:0} scn={val:1748675156950562001, v:0} tx_id={txid:11503239} prev=0x2b097ccdb4b8 next=(nil) modify_count=3 acc_checksum=4049869102 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678342993076000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1748678343092408000, v:0}, *this={this=0x2b097cccdba8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b097ccdb670 latest_compact_node=(nil) max_trans_version={val:1748678282963410001, v:0} max_trans_id=11577206 max_elr_trans_version={val:1748678282963410001, v:0} max_elr_trans_id=11577206 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748675156950562001, v:0} min_modify_scn={val:1748674856277973001, v:0}}) [2025-05-31 02:59:03.095136] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=48][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0980349e68 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577858} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2b08ee8cc5c0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678372994769, tablet_id:{id:309}, table_iter:0x2b08ee8caaf8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678372994769, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678343092408000, v:0}, tx_id:{txid:11577858}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11577858}, tx_desc:{this:0x2b0991b994a0, tx_id:{txid:11577858}, state:1, addr:"172.16.135.166:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1748678343092408, active_ts:1748678343092408, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678343124809, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1748678343092408000, v:0}, ref:2}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:03.095223] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11577858}, data:0x2b08ee8c94c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:03.095240] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.166", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11577858} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:03.095267] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=25][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.095278] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa6986f0, local_datums:0x2b09aa6986f0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa69857c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa698584, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa698592, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa69859a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09aa6985a2, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985b2, hex: B86877E769360600, int: 1748678343092408}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985ba, hex: B86877E769360600, int: 1748678343092408}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985c2, hex: 00504C0000000000, int: 5001216}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa6985ca, hex: 00504C0000000000, int: 5001216}]}) [2025-05-31 02:59:03.095367] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=54][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.095415] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A566E-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577858} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.096105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.096286] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:03.097751] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=132] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678342684433001, v:0}, new_max_version={val:1748678342684433001, v:0}, last_min_version={val:1748678342502560001, v:0}, last_max_version={val:1748678342502560001, v:0}, total_time_us=11359, delta_us=413315) [2025-05-31 02:59:03.097789] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678342684433001, v:0}, min_version={val:1748678342684433001, v:0}, max_version={val:1748678342684433001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:03.108809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.109832] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=24] insert successful (ret=0, tenant_id=1, file_type=4, data_size=5001216) [2025-05-31 02:59:03.117117] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=30] insert successful (ret=0, tenant_id=500, file_type=4, data_size=155648) [2025-05-31 02:59:03.119121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.120114] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=29][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b098034a2b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577860} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2b097cd10298 trans_version={val:1748676659812218001, v:0} scn={val:1748676659812218001, v:0} tx_id={txid:11519528} prev=0x2b097cd100e0 next=(nil) modify_count=3 acc_checksum=2562605422 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678342993076000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5463} write_epoch=3, snapshot_version={val:1748678343116221000, v:0}, *this={this=0x2b097ccce720 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b097cd10298 latest_compact_node=(nil) max_trans_version={val:1748678282985422000, v:0} max_trans_id=11577208 max_elr_trans_version={val:1748678282985422000, v:0} max_elr_trans_id=11577208 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748676659812218001, v:0} min_modify_scn={val:1748674856322487001, v:0}}) [2025-05-31 02:59:03.120166] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=49][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b098034a2b8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11577860} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2b08ee8cc5c0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678373019709, tablet_id:{id:309}, table_iter:0x2b08ee8caaf8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678373019709, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678343116221000, v:0}, tx_id:{txid:11577860}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11577860}, tx_desc:{this:0x2b0991b9b500, tx_id:{txid:11577860}, state:1, addr:"172.16.135.166:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1748678343116221, active_ts:1748678343116221, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678343149492, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1748678343116221000, v:0}, ref:2}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:03.120370] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=202][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11577860}, data:0x2b08ee8c94c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:03.120390] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.166", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11577860} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:03.120640] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=246][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.120654] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09a6dee6f0, local_datums:0x2b09a6dee6f0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee57c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09a6dee584, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee592, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee59a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09a6dee5a2, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee5b2, hex: BDC577E769360600, int: 1748678343116221}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee5ba, hex: BDC577E769360600, int: 1748678343116221}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee5c2, hex: 0020D30000000000, int: 13836288}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09a6dee5ca, hex: 0020D30000000000, int: 13836288}]}) [2025-05-31 02:59:03.120759] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.120812] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5672-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577860} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.126744] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:03.126802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4710, iops:20, norm_iops:5, bw:92624, [delay/us]:prepare:17, schedule:517, submit:121, rt:2715, total:3371) [2025-05-31 02:59:03.126820] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:03.127031] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=184] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4710, avg_iops:20, avg_bw:92611, [delay/us]: prepare:17, schedule:517, submit:121, device:2715, total:3371) [2025-05-31 02:59:03.127491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=450] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:03.127522] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4024, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:501, total:515) [2025-05-31 02:59:03.127634] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:03.127688] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4024, [delay/us]: prepare:0, schedule:0, submit:600000000, device:501, total:515) [2025-05-31 02:59:03.127710] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:03.127726] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:03.127735] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:03.127744] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:03.127752] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:03.127761] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=190kB/s, net_out=133kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:03.128007] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=231] successfully alloc block(macro_id={[ver=1,mode=0,seq=399438][2nd=2721]}) [2025-05-31 02:59:03.129432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.132038] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137289004}], scn_array=[cnt:1, {val:1748678343116221001, v:0}], curr_size=561, accum_size=9895, time_cost=10680) [2025-05-31 02:59:03.139666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.147690] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=77] insert successful (ret=0, tenant_id=1, file_type=1, data_size=13836288) [2025-05-31 02:59:03.151498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.158081] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=47] insert successful (ret=0, tenant_id=500, file_type=1, data_size=0) [2025-05-31 02:59:03.160384] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5678-0-0] [lt=47][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:03.160440] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5678-0-0] [lt=51][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:03.160461] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5678-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09aa61e690, local_datums:0x2b09aa61e690}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e51c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b09aa61e524, hex: 3137322E31362E3133352E313636, cstr: 172.16.135.166}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e532, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e53a, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09aa61e542, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e551, hex: 156878E769360600, int: 1748678343157781}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e559, hex: 156878E769360600, int: 1748678343157781}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e561, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09aa61e569, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:03.160646] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5678-0-0] [lt=91][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:03.160741] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [81063][TimerWK1_DiskUseReport][T1][YB42AC1087A6-00063665443A5678-0-0] [lt=43] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11577863} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:03.161723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.167360] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:124) [81063][TimerWK1_DiskUseReport][T0][Y0-0000000000000000-0-0] [lt=59] insert successful (ret=0, tenant_id=1, file_type=3, data_size=0) [2025-05-31 02:59:03.172262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.182449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.192668] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.208157] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.218309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.227313] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:03.231230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.241493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.245363] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:03.246731] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=912253, clean_num=31457) [2025-05-31 02:59:03.252040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.256618] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83086][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3877-0-0] [lt=31][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525392438785, cache_obj->added_lc()=false, cache_obj->get_object_id()=53652, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:03.258195] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=60] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:03.262576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.273236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.283521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.289652] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=98, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:03.289707] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=22, avg_log_batch_cnt=1, total_group_log_size=11752, avg_group_log_size=534) [2025-05-31 02:59:03.289729] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=11752) [2025-05-31 02:59:03.289744] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=22, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:03.296070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.300679] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=6] get number(ret=0, service_type_=0, range=1, base_id=1748678343297932000, start_id=1748678343297932000, end_id=1748678343297932001) [2025-05-31 02:59:03.301698] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137290833}], scn_array=[cnt:1, {val:1748678343288669000, v:0}], curr_size=589, accum_size=11752, time_cost=11686) [2025-05-31 02:59:03.301810] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=111] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=11752) [2025-05-31 02:59:03.302014] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=22, avg_log_gen_to_freeze_time=77, avg_log_gen_to_submit_time=198, avg_log_submit_to_flush_time=5644, avg_log_submit_to_slide_time=6361) [2025-05-31 02:59:03.302341] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=11752, hit_cnt=22, read_cnt=22, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:03.304035] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC108714-000636671EBE41F7-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1748678343302798000, start_id=1748678343302798000, end_id=1748678343302798001) [2025-05-31 02:59:03.306724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=352][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.317082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.328038] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.328211] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=27] rpc statistics(total_trans_msg_count=17, total_batch_msg_count=0) [2025-05-31 02:59:03.338315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.348710] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.358403] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.359152] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.361589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.361629] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=72] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.361657] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.369464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.376138] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=25] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:03.378330] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=53] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:03.378463] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=45] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678342188585000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:03.378534] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=67] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678342188585000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678342188585000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:59:03.382444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.395188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.405574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.415980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.427566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=102] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:03.427941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.438311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.447226] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:03.449810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.450441] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ts response task statistics(alloc_count=486386, free_count=243192) [2025-05-31 02:59:03.450545] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=89] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:03.450601] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678342948539], gts:1748678342985294000, latest_srr:[mts=1748678343450565]}) [2025-05-31 02:59:03.460140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.471475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.482435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.483150] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=4086, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:03.483251] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=85] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=652, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:03.483419] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=129] [PALF STAT FLUSH CB COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:03.491239] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA65-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653509426116943359e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.722015445862007709e+03) [2025-05-31 02:59:03.498462] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83076][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3879-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525392680631, cache_obj->added_lc()=false, cache_obj->get_object_id()=53653, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:03.498658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.509433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.517378] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:03.519801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.532552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.542864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.548353] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:03.553166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.563579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.574369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.586279] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A6E-0-0] [lt=42] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:03.587312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.591254] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591352] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591377] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591787] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591912] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.591947] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:03.597591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.609765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.619264] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=44] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:03.620117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.629953] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:03.630812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.641326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.647826] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:03.651603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.660151] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83080][T1_L0_G0][T1][YB42AC10871E-000636654E089A38-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525392842320, cache_obj->added_lc()=false, cache_obj->get_object_id()=53654, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:03.661923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.672156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.672455] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81744][T1_TimerWK3_TabletShell][T1][Y0-0000000000000000-0-0] [lt=39] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:03.672489] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81744][T1_TimerWK3_TabletShell][T1][Y0-0000000000000000-0-0] [lt=22] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=995) [2025-05-31 02:59:03.682990] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.690107] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA76-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:03.693221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.698418] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=70] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:03.703443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.713705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.724086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.734339] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.736040] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E93-0-0] [lt=34] submit log success(service_type_=0, last_id=1748678360364653000, limited_id=1748678360364653000) [2025-05-31 02:59:03.736093] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1E93-0-0] [lt=52] submit log with lock success(service_type_=0, last_id=1748678360364653000, limited_id=1748678360364653000) [2025-05-31 02:59:03.740532] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=22] update id meta(ret=0, service_type=0, limited_id=1748678360364653000, latest_scn={val:1748678343734669001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:59:03.740757] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=223] handle submit callback(service_type_=0, success=true, used_ts=4720, log_ts={val:1748678343734669001, v:0}, limited_id=1748678360364653000) [2025-05-31 02:59:03.740787] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=29] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:59:03.740798] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3876-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:59:03.744620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.755745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.766153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.776745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.787270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.797563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.807922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.818473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.828716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.830132] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:03.839352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.848356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=122] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:03.849611] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.851463] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4488-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:03.859930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.870145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.881792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.884604] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1145-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=13117) [2025-05-31 02:59:03.894607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.898028] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1146-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=11, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:03.898055] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1146-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=6484, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:03.905601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.905647] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:03.905701] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [23361][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=60] worker thread exit(tenant_->id()=1, workers_.get_size()=4) [2025-05-31 02:59:03.915910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.926369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.933723] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=31] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:59:03.938018] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.939466] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=41] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:03.948709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.951677] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678343450845], gts:1748678343488916000, latest_srr:[mts=1748678343951648]}) [2025-05-31 02:59:03.959320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.969755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.980073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.993080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:03.993606] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=49] PNIO [ratelimit] time: 1748678343993602, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006475, add_bytes: 0 [2025-05-31 02:59:04.003320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.013573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.021293] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=43] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:04.025130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.030336] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:04.035756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.046312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.050945] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:04.053028] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:04.055268] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1748678344055265, bytes: 7133592242, bw: 0.190300 MB/s, add_ts: 1008196, add_bytes: 201180 [2025-05-31 02:59:04.059414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=141][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.064282] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=26] current monitor number(seq_=-1) [2025-05-31 02:59:04.064407] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=16] gc stale ls task succ [2025-05-31 02:59:04.069785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.078637] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81744][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:04.080398] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=136][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.086087] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:04.093006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.094012] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] no need to update min/max version(new_version={val:1748678343687160000, v:0}, min_version={val:1748678343687160000, v:0}, max_version={val:1748678343687160000, v:0}) [2025-05-31 02:59:04.099104] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=15] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:04.106318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.116717] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.127199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.139953] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF2-0-0] [lt=38] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:04.140297] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:04.140418] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4369, iops:15, norm_iops:3, bw:64659, [delay/us]:prepare:22, schedule:1001, submit:256, rt:3018, total:4298) [2025-05-31 02:59:04.140464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.140441] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:04.140648] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=189] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4369, avg_iops:15, avg_bw:64655, [delay/us]: prepare:22, schedule:1001, submit:256, device:3018, total:4298) [2025-05-31 02:59:04.140663] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:04.140748] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4042, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:622, total:633) [2025-05-31 02:59:04.140829] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:04.141070] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=148] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4041, [delay/us]: prepare:0, schedule:0, submit:600000000, device:622, total:633) [2025-05-31 02:59:04.141190] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:04.141216] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:04.141225] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:04.141234] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:04.141243] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:04.141253] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=180kB/s, net_out=114kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:04.141274] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399439][2nd=3003]}) [2025-05-31 02:59:04.151123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.161426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.166179] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1748678344166173, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:59:04.166283] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=101] snapshot_traffic_info once [2025-05-31 02:59:04.168412] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206D-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:04.168453] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206D-0-0] [lt=31] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=42) [2025-05-31 02:59:04.171700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.184125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.185304] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:76K/17:935535 disk:76K/17:935535 t1m2:76K/17:935535 t1m2g0:76K/17:935535 |q0: 0 |q1: 0 [2025-05-31 02:59:04.200615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.206403] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678343891274000, v:0}, new_max_version={val:1748678343891274000, v:0}, last_min_version={val:1748678343687160000, v:0}, last_max_version={val:1748678343687160000, v:0}, total_time_us=10336, delta_us=315125) [2025-05-31 02:59:04.206465] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678343891274000, v:0}, min_version={val:1748678343891274000, v:0}, max_version={val:1748678343891274000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:04.215474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.226462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.230938] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:04.237201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.247438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.252300] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:04.258536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.266716] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A387B-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525393448860, cache_obj->added_lc()=false, cache_obj->get_object_id()=53655, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:04.273084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.284019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.293667] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:04.294282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.307459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.317618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.327973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.337476] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=54] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:04.337505] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=29] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:04.337590] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F45-0-0] [lt=15] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:04.337616] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F45-0-0] [lt=21] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:04.338536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.348707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.350893] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=56] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:04.358587] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.359071] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.359207] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.359729] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.359767] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.369491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.380089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.386745] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:04.386814] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5017, avg_group_log_size=385) [2025-05-31 02:59:04.386840] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5484) [2025-05-31 02:59:04.386961] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=117] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:04.389010] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5017) [2025-05-31 02:59:04.390529] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC114A-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=107, avg_log_gen_to_submit_time=215, avg_log_submit_to_flush_time=3833, avg_log_submit_to_slide_time=4647) [2025-05-31 02:59:04.390740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.394031] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5017, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:04.402159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.412448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.422707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.428216] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=31] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:59:04.432162] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:04.433650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.443896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.452205] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:04.452276] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=2, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:04.454176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.456195] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:04.465387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.475662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.486140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.496311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.498423] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678344401706000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678344498367}, version_delta=96714) [2025-05-31 02:59:04.507538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.518139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.521598] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-05-31 02:59:04.528719] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.539651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.549947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.553351] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:04.560321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.570699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.581103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.588182] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3318, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:04.588223] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=744, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:04.588288] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=211] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=40, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:04.589951] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.591635] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.591798] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.592144] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.592177] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.592232] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.592304] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:04.592317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.600801] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:04.600935] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:04.600986] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:04.601011] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:04.602326] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA74-0-0] [lt=38] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.581508922576904297e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=279, Throttle Trigger(MB)=167, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.863926391757209785e+03) [2025-05-31 02:59:04.604603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.614107] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=41472) [2025-05-31 02:59:04.617683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.620543] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F7E3EC0-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1748678344619727000, start_id=1748678344619727000, end_id=1748678344619727001) [2025-05-31 02:59:04.629561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.631974] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D2-0-0] [lt=17] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:04.632363] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:04.641726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.652822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.657260] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:04.663244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.673660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.688455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.698991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.710147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.720775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.730328] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA77-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:04.731617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.737509] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:338) [83087][T1_L0_G0][T1][YB42AC10871E-0006366543F8A3F6-0-0] [lt=42][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:11577877}, ref_cnt:1, state:"RUNNING", commit_version:{val:1748678344726811000, v:0}, start_scn:{val:1748678344726811000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1748678374635082, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678344726811000, v:0}, tx_id:{txid:11577878}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, data_trans_id_:{txid:11577877}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1748678344726811000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1748678344726811000, v:0}}) [2025-05-31 02:59:04.738316] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83087][T1_L0_G0][T1][YB42AC10871E-0006366543F8A3F6-0-0] [lt=100] get number(ret=0, service_type_=0, range=1, base_id=1748678344737182000, start_id=1748678344737182000, end_id=1748678344737182001) [2025-05-31 02:59:04.742043] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:04.742079] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] temporary file module use no blocks [2025-05-31 02:59:04.742090] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:04.742105] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:04.742121] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525392923834, flush_io_finished_ret_:0, flush_io_finished_round_:19912, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:04.742169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.752966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.763242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.773508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.784962] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.795584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.806299] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1748678344803293000, start_id=1748678344803293000, end_id=1748678344803293001) [2025-05-31 02:59:04.808505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.812947] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=107] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:04.819080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.829383] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.835652] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:04.839703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.850300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.855248] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4489-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:04.860728] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:04.863667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.873995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.878958] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=32] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19335675904, fast_delete_log_mode=false) [2025-05-31 02:59:04.879099] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-05-31 02:59:04.879128] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=19] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:59:04.879152] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:04.886780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.897678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.903841] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=134] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678344619727000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=284094, in_cluster_service=true, cluster_version={val:1748678344600649000, v:0}, min_cluster_version={val:1748678344600649000, v:0}, max_cluster_version={val:1748678344600649000, v:0}, get_cluster_version_err=0, cluster_version_delta=303172, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195292, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=335849063, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:04.908503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=380][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.908558] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:04.909027] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [23779][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=27] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-05-31 02:59:04.919976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.920462] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3D-0-0] [lt=39] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678344918281, keepalive_ts:-1}) [2025-05-31 02:59:04.920499] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3D-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678344918281, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 03:59:04.921064] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [25121][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:04.921246] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-05-31 02:59:04.931725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.942949] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=36] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:04.945778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.953240] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1748678344952931000, start_id=1748678344952931000, end_id=1748678344952931001) [2025-05-31 02:59:04.955293] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=31] success to update min reserved snapshot(reserved_snapshot=1748676542188585000, duration=1800, snapshot_gc_ts=1748678342188585000) [2025-05-31 02:59:04.957573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.959028] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5199) [2025-05-31 02:59:04.960043] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1152-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=21, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:04.960103] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1152-0-0] [lt=56] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5061, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:04.961030] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=60] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1748676542188585000) [2025-05-31 02:59:04.961080] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=47] update reserved snapshot finished(cost_ts=5796, reserved_snapshot=1748676542188585000) [2025-05-31 02:59:04.967819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.978124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.989264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:04.999617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.000807] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1748678345000801, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007199, add_bytes: 0 [2025-05-31 02:59:05.003977] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=177] no need to update min/max version(new_version={val:1748678344600649000, v:0}, min_version={val:1748678344600649000, v:0}, max_version={val:1748678344600649000, v:0}) [2025-05-31 02:59:05.010201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=350][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.021334] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.021825] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=135] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:05.031603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.037740] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:05.042556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.046602] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=164] ====== check clog disk timer task ====== [2025-05-31 02:59:05.046679] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=68] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:05.052830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.054654] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:05.061527] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:05.062231] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1748678345062227, bytes: 7133760790, bw: 0.159629 MB/s, add_ts: 1006962, add_bytes: 168548 [2025-05-31 02:59:05.064296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.064808] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:59:05.075526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.086514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.087534] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:05.098932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:05.099355] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [25162][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:05.099578] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:05.100245] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread created(tenant_->id()=1, group_id_=1, token=4) [2025-05-31 02:59:05.101599] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81740][T1_TimerWK0_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:05.110436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.112331] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:68K/15:526876 disk:68K/15:526876 t1m2:68K/15:526876 t1m2g0:68K/15:526876 |q0: 0 |q1: 0 [2025-05-31 02:59:05.121034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.131398] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB94708-0-0] [lt=12][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:717859798, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678345161607, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1385321, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678345142044, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678345131391, enqueue_timestamp_:0, request_arrival_time_:1748678345131391, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:05.131486] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB94708-0-0] [lt=66][errcode=-5150] deliver rpc request fail(&req=0x2b099d616098, ret=-5150) [2025-05-31 02:59:05.131760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.141994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.153964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.160471] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:05.160593] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=84] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:15, norm_iops:4, bw:68250, [delay/us]:prepare:17, schedule:553, submit:176, rt:2412, total:3159) [2025-05-31 02:59:05.160614] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:05.160673] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:15, avg_bw:68264, [delay/us]: prepare:17, schedule:553, submit:176, device:2412, total:3159) [2025-05-31 02:59:05.160686] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:05.160713] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4015, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:344, total:362) [2025-05-31 02:59:05.160738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:05.160766] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4016, [delay/us]: prepare:0, schedule:0, submit:600000000, device:344, total:362) [2025-05-31 02:59:05.160784] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:05.160799] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:05.160808] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:05.160816] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:05.160824] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:05.160833] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=152kB/s, net_out=144kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:05.160854] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399440][2nd=2298]}) [2025-05-31 02:59:05.164339] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.174613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.184986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.195230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.205457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.220697] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.232253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.246045] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=470] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=346016, replace_num=15728) [2025-05-31 02:59:05.246267] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=218] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:05.247104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.260737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.262203] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:05.263731] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=943710, clean_num=31457) [2025-05-31 02:59:05.272311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.284281] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=209][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.286592] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83076][T1_L0_G0][T1][YB42AC1087A6-000636654C3A387D-0-0] [lt=105][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525394468760, cache_obj->added_lc()=false, cache_obj->get_object_id()=53656, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:05.294803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.305218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=228][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.306618] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=81] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678345105798001, v:0}, base_version={val:1748678344737182000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678345105798001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678345306597, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678345105798001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678345310847, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678345105798001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678345233667, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678340307059000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678345241670, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678340631972000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678345306566, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678340238749000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678345249302, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:59:05.315481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.320778] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137301587}], scn_array=[cnt:1, {val:1748678345306580000, v:0}], curr_size=589, accum_size=5737, time_cost=10683) [2025-05-31 02:59:05.321597] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678345105798001, v:0}, new_max_version={val:1748678345105798001, v:0}, last_min_version={val:1748678344737182000, v:0}, last_max_version={val:1748678344737182000, v:0}, total_time_us=14914, delta_us=215797) [2025-05-31 02:59:05.321661] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678345105798001, v:0}, min_version={val:1748678345105798001, v:0}, max_version={val:1748678345105798001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:05.327027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:05.329623] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [25226][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:05.329678] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] worker thread created(tenant_->id()=1, group_id_=1, token=5) [2025-05-31 02:59:05.339898] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.352713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.358627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.358675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.358966] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.359416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.362852] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.364678] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA7B-0-0] [lt=100][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525394546850, cache_obj->added_lc()=false, cache_obj->get_object_id()=53657, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:05.369181] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=38][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:05.373129] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=14] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:05.373259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.384104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.394680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.405012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.415750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.426003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.436835] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.446641] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:05.448355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=144][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.454237] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=136] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=661, get_gts_with_stc_cnt=58, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=46, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:05.454275] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678345454200], gts:1748678345451280000, latest_srr:[mts=1748678345454200]}) [2025-05-31 02:59:05.454295] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:05.458618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.464326] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:05.469111] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.480021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.492147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.499113] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=98, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:05.499160] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=15, avg_log_batch_cnt=1, total_group_log_size=5859, avg_group_log_size=390) [2025-05-31 02:59:05.499182] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5859) [2025-05-31 02:59:05.499197] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:05.501425] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=100] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5859) [2025-05-31 02:59:05.502368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:05.502718] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [25226][T1_L0_G1][T1][YB42AC10871E-0006366549CC1159-0-0] [lt=128] init point thread id with(&point=0x55c0908d1640, idx_=650, point=[thread id=25226, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=25226) [2025-05-31 03:59:05.502907] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [25226][T1_L0_G1][T1][YB42AC10871E-0006366549CC1159-0-0] [lt=34] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=15, avg_log_gen_to_freeze_time=166, avg_log_gen_to_submit_time=284, avg_log_submit_to_flush_time=4289, avg_log_submit_to_slide_time=5037) [2025-05-31 02:59:05.505483] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5859, hit_cnt=15, read_cnt=15, hit rate=1.000000000000000000e+00) [2025-05-31 03:59:05.506361] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [25292][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:05.506660] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=168] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-05-31 02:59:05.516909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.522190] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:05.527132] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.537747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.541288] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] ====== checkpoint timer task ====== [2025-05-31 02:59:05.542133] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=37] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:05.542208] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=73] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:05.542573] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:05.542790] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=203] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:05.543223] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=423] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:05.543240] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1748678343734669001, v:0}) [2025-05-31 02:59:05.543248] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:05.543256] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:05.543270] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:05.543291] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:05.543350] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=54] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:05.543376] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:05.543399] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:05.543411] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:05.550084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.555787] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:05.560584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.562137] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83085][T1_L0_G0][T1][YB42AC108714-000636671E6D5136-0-0] [lt=78][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525394744260, cache_obj->added_lc()=false, cache_obj->get_object_id()=53658, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:05.571447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.585406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.591091] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.591145] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.591174] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.591200] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.591226] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=82] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.594033] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.594077] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:05.596263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.607020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.620465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.630765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.641094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.649641] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:05.653521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.663748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.667384] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:05.674065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.685813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.696106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.706256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.706735] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=3808, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:05.706774] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=518, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:05.706813] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=115] [PALF STAT FLUSH CB COST TIME](cur_stat_count=15, stat_interval=1000000, avg (count/cost)=54, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:05.710021] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA83-0-0] [lt=60] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653509426116943359e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.722015445862007709e+03) [2025-05-31 02:59:05.710676] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x55c090a74fc0, last_check_time:1748678345710672, last_idle_time:78781398326, last_thread_count:4}, *pool={name:TimerWK, this:0x55c090a74fc0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:78781398326, tenant_id:500}, qsize=-4, current_idle_time=78781398326) [2025-05-31 02:59:05.710711] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=35] [thread pool stat](pool_stat={pool:0x2b08f13edcf0, last_check_time:1748678345710710, last_idle_time:42489891912, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13edcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:42489891912, tenant_id:500}, qsize=-1, current_idle_time=42489891912) [2025-05-31 02:59:05.710724] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] [thread pool stat](pool_stat={pool:0x2b08f13eeaf0, last_check_time:1748678345710723, last_idle_time:19961217742, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13eeaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:19961217742, tenant_id:500}, qsize=-1, current_idle_time=19961217742) [2025-05-31 02:59:05.710734] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2b08efdfee70, last_check_time:1748678345710734, last_idle_time:77291123307, last_thread_count:4}, *pool={name:TimerWK, this:0x2b08efdfee70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:77291123307, tenant_id:1}, qsize=-4, current_idle_time=77291123307) [2025-05-31 02:59:05.710744] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2b08f13ef1f0, last_check_time:1748678345710744, last_idle_time:2788633703, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2788633703, tenant_id:500}, qsize=-1, current_idle_time=2788633703) [2025-05-31 02:59:05.710754] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2b08f13ef8f0, last_check_time:1748678345710754, last_idle_time:2788191037, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef8f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2788191037, tenant_id:500}, qsize=-1, current_idle_time=2788191037) [2025-05-31 02:59:05.717906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.728044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.738315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.742243] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:05.742336] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=84] temporary file module use no blocks [2025-05-31 02:59:05.742349] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:05.742365] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:05.742406] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=39] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525393924335, flush_io_finished_ret_:0, flush_io_finished_round_:19913, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:05.748563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.757058] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83094][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA78-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:05.759001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.769986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.780568] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.792070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.802634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.812924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.823249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.833494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.843956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.850336] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:05.854245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.858273] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448A-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:05.864746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.868397] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:05.874948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.885238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.896623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.904900] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:05.904936] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] try gc retain ctx [2025-05-31 02:59:05.904944] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-31 02:59:05.904992] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678345706851000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678335722829001, v:0}, keep_alive_scn_:{val:1748678340674232001, v:0}, update_ts_:1748678345904791}) [2025-05-31 02:59:05.905685] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:05.907400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.908062] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] no need to update min/max version(new_version={val:1748678345451280000, v:0}, min_version={val:1748678345451280000, v:0}, max_version={val:1748678345451280000, v:0}) [2025-05-31 02:59:05.908171] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=101] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:05.917712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.928244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.938819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.943746] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=45] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:05.951330] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=33] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:59:05.951523] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=173][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.957067] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:05.961759] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.972329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.982815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:05.993134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.004967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.008437] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=353] PNIO [ratelimit] time: 1748678346008432, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007631, add_bytes: 0 [2025-05-31 02:59:06.011094] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC115E-0-0] [lt=19] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3739) [2025-05-31 02:59:06.014741] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=30, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:06.014788] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5536, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:06.015275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.024227] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:06.024315] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [25162][T1_L0_G1][T1][YB42AC1087A7-00063665517F5599-0-0] [lt=195] init point thread id with(&point=0x55c0908d0640, idx_=586, point=[thread id=25162, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=25162) [2025-05-31 02:59:06.025798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.036908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.048085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.052701] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:06.055947] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=81] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:06.060027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.064574] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678346064569, bytes: 7133911395, bw: 0.143293 MB/s, add_ts: 1002342, add_bytes: 150605 [2025-05-31 02:59:06.065189] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2025-05-31 02:59:06.065274] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:59:06.068991] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:06.070249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.082919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.087923] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:06.094546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.100602] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:06.104686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.114913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.127494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.137653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.148133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.157232] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B34-0-0] [lt=11] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:06.158550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.165005] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:06.165080] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:9, norm_iops:2, bw:36699, [delay/us]:prepare:22, schedule:597, submit:386, rt:3312, total:4318) [2025-05-31 02:59:06.165101] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:06.165158] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:9, avg_bw:36699, [delay/us]: prepare:22, schedule:597, submit:386, device:3312, total:4318) [2025-05-31 02:59:06.165172] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:06.165198] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4077, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:381, total:395) [2025-05-31 02:59:06.165223] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:06.165251] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4077, [delay/us]: prepare:0, schedule:0, submit:600000000, device:381, total:395) [2025-05-31 02:59:06.165269] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:06.165283] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:06.165291] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:06.165299] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:06.165308] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:06.165316] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=143kB/s, net_out=94kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:06.165336] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399441][2nd=2580]}) [2025-05-31 02:59:06.168229] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=27] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:06.168276] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=36] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:06.168293] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=959) [2025-05-31 02:59:06.169030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.179983] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206E-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:06.180039] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206E-0-0] [lt=46] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=58) [2025-05-31 02:59:06.180214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.191456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.202243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.208955] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:40K/10:507991 disk:40K/10:507991 t1m2:40K/10:507991 t1m2g0:40K/10:507991 |q0: 0 |q1: 0 [2025-05-31 02:59:06.214585] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.214812] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [25292][T1_L0_G1][T1][YB42AC1087A7-00063665517F559A-0-0] [lt=40] init point thread id with(&point=0x55c0908d26c0, idx_=716, point=[thread id=25292, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=25292) [2025-05-31 03:59:06.217443] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [25572][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:06.218136] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:06.231954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.236599] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=93] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:06.242575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.253328] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:06.255935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.268233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.269567] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:06.278600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.291487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.294059] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A387F-0-0] [lt=60][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525395476228, cache_obj->added_lc()=false, cache_obj->get_object_id()=53659, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:06.302341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.309783] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4209-0-0] [lt=17] get number(ret=0, service_type_=0, range=1, base_id=1748678346309528000, start_id=1748678346309528000, end_id=1748678346309528001) [2025-05-31 02:59:06.314780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.327576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.339073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.340385] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=51] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:06.340410] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:06.340540] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F46-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:06.340566] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F46-0-0] [lt=22] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:06.349985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.358850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.359142] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.359198] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.359244] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.360725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.369953] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=67] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:06.370918] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.382002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.396165] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=6] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:06.399300] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=77] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:06.399453] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=29] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678342188585000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:06.402098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.413505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.422820] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:06.423741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.427635] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137305133}], scn_array=[cnt:1, {val:1748678346211996001, v:0}], curr_size=122, accum_size=2957, time_cost=18935) [2025-05-31 02:59:06.431755] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=217] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:06.433758] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=34] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:06.435195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.435720] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=20] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:06.435740] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=19] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:06.437527] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:06.437594] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:06.437690] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:06.437729] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=26] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:06.438365] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=629] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:06.438408] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=33] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:06.438420] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:06.438443] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:06.438464] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:06.441044] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:06.441076] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=30] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:06.441086] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:06.441121] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:06.441134] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:06.441143] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:06.441156] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:06.441172] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDC-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:06.441534] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137305255}], scn_array=[cnt:1, {val:1748678346427046000, v:0}], curr_size=589, accum_size=3546, time_cost=13302) [2025-05-31 02:59:06.442396] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678346012944001, v:0}, new_max_version={val:1748678346012944001, v:0}, last_min_version={val:1748678345956259000, v:0}, last_max_version={val:1748678345956259000, v:0}, total_time_us=17415, delta_us=429450) [2025-05-31 02:59:06.442432] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678346012944001, v:0}, min_version={val:1748678346012944001, v:0}, max_version={val:1748678346012944001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:06.447609] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.453622] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:06.458399] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=33] ts response task statistics(alloc_count=486458, free_count=243228) [2025-05-31 02:59:06.458434] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:06.458473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.458486] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678345956457], gts:1748678345998602000, latest_srr:[mts=1748678346458449]}) [2025-05-31 02:59:06.469069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.470546] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:06.479418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=173][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.489693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.500295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.511240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.518593] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3881-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525395700761, cache_obj->added_lc()=false, cache_obj->get_object_id()=53660, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:06.519903] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3883-0-0] [lt=53][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:06.520991] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3885-0-0] [lt=25][errcode=-4201] hashtable not init [2025-05-31 02:59:06.521263] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1748678346520847000, start_id=1748678346520847000, end_id=1748678346520847001) [2025-05-31 02:59:06.521471] INFO [LIB] stat (utility.h:1211) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:06.521501] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4112, avg_group_log_size=373) [2025-05-31 02:59:06.521520] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3668) [2025-05-31 02:59:06.521535] INFO [LIB] stat (utility.h:1211) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:06.521642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.524187] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4112) [2025-05-31 02:59:06.524508] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=63] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:06.526657] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1164-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=135, avg_log_gen_to_submit_time=235, avg_log_submit_to_flush_time=6687, avg_log_submit_to_slide_time=7034) [2025-05-31 02:59:06.527140] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4112, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:06.532136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.542965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.553240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.556133] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:06.565432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.577390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.590107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.590287] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.594069] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83094][T1_L0_G0][T1][YB42AC108714-000636671E2D3A6F-0-0] [lt=46] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:06.594991] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.595033] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.595059] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.595084] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.595109] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.595250] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:06.600688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.610871] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.621409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.633003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.639773] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D3-0-0] [lt=76] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:06.643485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:06.643643] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [25572][T1_L0_G1][T1][YB42AC1087A7-00063665517F559F-0-0] [lt=4898] init point thread id with(&point=0x55c0908d6cc0, idx_=996, point=[thread id=25572, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=25572) [2025-05-31 02:59:06.654047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.654285] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:06.665145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.671603] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81743][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=45] ====== traversal_flush timer task ====== [2025-05-31 02:59:06.672061] INFO [MDS] flush (mds_table_mgr.cpp:184) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EDF-0-0] [lt=26] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678346623414000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:06.672225] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EDF-0-0] [lt=155] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:06.672351] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:06.675407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.687659] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.698404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.710586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.721293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.730274] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:06.731516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.732203] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F31-0-0] [lt=45] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=46) [2025-05-31 02:59:06.732313] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F31-0-0] [lt=90] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653509426116943359e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.722015445862007709e+03) [2025-05-31 02:59:06.732542] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1748678346731943000, start_id=1748678346731943000, end_id=1748678346731943001) [2025-05-31 02:59:06.736192] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5673, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:06.736244] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=564, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:06.736691] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=79, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:06.743055] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=68] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:06.743095] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-05-31 02:59:06.743106] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:06.743122] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:06.743138] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525394924660, flush_io_finished_ret_:0, flush_io_finished_round_:19914, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:06.743266] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.754247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.764531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.777657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.777921] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=26] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:06.777952] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=26] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:06.777965] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:06.777976] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:06.781280] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=8] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:06.782090] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=804] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:06.787945] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA79-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:06.788434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.798959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.806118] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3E4-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525395988287, cache_obj->added_lc()=false, cache_obj->get_object_id()=53661, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:06.813693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.824120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.835281] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.837180] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [25121][T1_L0_G1][T1][YB42AC10871E-0006366549CC1167-0-0] [lt=92] init point thread id with(&point=0x55c0908cfc00, idx_=545, point=[thread id=25121, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=25121) [2025-05-31 02:59:06.846634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.854616] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:06.856882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.860799] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448B-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:06.867238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.872925] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:06.877518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.887750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.892442] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=52] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:06.897340] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:06.897436] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=42] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52599580, tenant_id=1) [2025-05-31 02:59:06.897464] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=19] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:06.898262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.908929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.919115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.929374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.941738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.943915] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=43] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:06.953061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=415][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.960316] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678346458706], gts:1748678346494590000, latest_srr:[mts=1748678346960275]}) [2025-05-31 02:59:06.963811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.976294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.987025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:06.998137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.008461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.016237] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678347016231, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007799, add_bytes: 0 [2025-05-31 02:59:07.018662] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1168-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4424) [2025-05-31 02:59:07.018909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.021664] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:07.021694] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7250, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:07.025552] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=51] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:07.036657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.037021] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81065][TimerWK3_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=24] runTimer to refresh locality_info(ret=0) [2025-05-31 02:59:07.039626] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=123] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:59:07.039718] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=78] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:59:07.039735] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:59:07.039756] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:59:07.039807] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=49] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:59:07.039833] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=10] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:59:07.039849] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=17] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:59:07.039859] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:59:07.046933] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-05-31 02:59:07.046982] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=41] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:07.048146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.051218] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:07.054981] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:07.056346] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:07.058490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.065666] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=41] current monitor number(seq_=-1) [2025-05-31 02:59:07.065971] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=19] gc stale ls task succ [2025-05-31 02:59:07.068663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.069260] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1748678347069255, bytes: 7134125627, bw: 0.203355 MB/s, add_ts: 1004686, add_bytes: 214232 [2025-05-31 02:59:07.073457] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:07.078886] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.080107] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA0-0-0] [lt=43] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678347114914, keepalive_ts:-1}) [2025-05-31 02:59:07.080145] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA0-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678347114914, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:07.088133] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:07.089042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.100065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.100852] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:07.112824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.123319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.136893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.147315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.157923] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF3-0-0] [lt=46][errcode=-4006] not init(ret=-4006) [2025-05-31 02:59:07.157979] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF3-0-0] [lt=48] finish scan map holding items(scan_cnt=0) [2025-05-31 02:59:07.163568] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF4-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:07.163687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.173986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.176396] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:07.176453] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44548, [delay/us]:prepare:13, schedule:149, submit:234, rt:4818, total:5215) [2025-05-31 02:59:07.176471] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:07.176608] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=121] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44545, [delay/us]: prepare:13, schedule:149, submit:234, device:4818, total:5215) [2025-05-31 02:59:07.176623] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:07.176650] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4049, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:724, total:738) [2025-05-31 02:59:07.176676] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:07.176706] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4049, [delay/us]: prepare:0, schedule:0, submit:600000000, device:724, total:738) [2025-05-31 02:59:07.176724] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:07.176738] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:07.176747] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:07.176755] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:07.176763] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:07.176772] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=214kB/s, net_out=123kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:07.176793] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399442][2nd=2452]}) [2025-05-31 02:59:07.184390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.194786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.205114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.216376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.216452] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:07.217089] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [25292][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:07.221359] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=48] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:40K/9:146393 disk:40K/9:146393 t1m2:40K/9:146393 t1m2g0:40K/9:146393 |q0: 0 |q1: 0 [2025-05-31 02:59:07.227583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.244046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.250616] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1EA8-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678347248276000, start_id=1748678347248276000, end_id=1748678347248276001) [2025-05-31 02:59:07.254402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.255374] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:07.265223] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.272798] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33CB-0-0] [lt=47][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525396454967, cache_obj->added_lc()=false, cache_obj->get_object_id()=53662, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.274610] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:07.276115] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=62] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=975167, clean_num=31457) [2025-05-31 02:59:07.278566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.288807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.301946] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3887-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525396484114, cache_obj->added_lc()=false, cache_obj->get_object_id()=53663, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.302265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.312488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.322757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.333078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.343374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.353801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.360168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=197] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.360231] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.360262] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.360289] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.364317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.374562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.385461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.396041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.406605] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.418778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.429006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.440713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.447233] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678347024417001, v:0}, new_max_version={val:1748678347024417001, v:0}, last_min_version={val:1748678346959410001, v:0}, last_max_version={val:1748678346959410001, v:0}, total_time_us=9282, delta_us=422812) [2025-05-31 02:59:07.447305] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678347024417001, v:0}, min_version={val:1748678347024417001, v:0}, max_version={val:1748678347024417001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:07.451505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.457049] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=69] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=361744, replace_num=15728) [2025-05-31 02:59:07.457094] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:07.460954] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:07.462564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.474433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.476783] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:07.484690] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.497371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.507569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.518499] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3889-0-0] [lt=29][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525396700670, cache_obj->added_lc()=false, cache_obj->get_object_id()=53664, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.519252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.526187] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:07.529529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.540240] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.551771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.561051] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:07.564480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.575293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.585724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.592365] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=70] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592438] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592485] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592512] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592539] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.592662] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:07.593975] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A567E-0-0] [lt=25] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:07.596152] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.600019] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:07.600060] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.600099] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.600123] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:07.606553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.613329] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] start to print blacklist info [2025-05-31 02:59:07.613756] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] ls blacklist refresh finish(cost_time=40661) [2025-05-31 02:59:07.621602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.628453] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=124] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:07.628504] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3980, avg_group_log_size=398) [2025-05-31 02:59:07.628526] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4424) [2025-05-31 02:59:07.628541] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:07.632995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.634319] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3980) [2025-05-31 02:59:07.634615] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=4299, avg_log_submit_to_slide_time=5367) [2025-05-31 02:59:07.635673] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3980, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:07.643340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.653703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.657340] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:07.664199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.675050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=381][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.679695] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:07.685337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.688765] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=60] new mysql sessid created(conn.sessid_=3221738425, support_ssl=false) [2025-05-31 02:59:07.688826] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221738425, sess.client_addr_="172.16.135.166:43888") [2025-05-31 02:59:07.688844] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2b08ff896450, session_id:3221738425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:141}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:07.689562] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81569][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=134] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:59:07.689586] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [81569][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=22] mysql login queue(mysql_queue->queue_.size()=1) [2025-05-31 02:59:07.692411] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCD-0-0] [lt=27] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221738425, conn.client_addr_port_=43888, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221738425, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221738425, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-0006366544F9FFCD-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82989, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418658276475623, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:2853, fixup_index:-1, fixup_ash_buffer:{this:0x2b092b3fb6f8, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:59:07.694473] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCD-0-0] [lt=88][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:59:07.694586] INFO [SERVER] process (obmp_connect.cpp:504) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCD-0-0] [lt=41] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221738425, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:59:07.695966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.696313] INFO [SERVER] process (obmp_init_db.cpp:44) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A388B-0-0] [lt=35] init db(db_name=oceanbase) [2025-05-31 02:59:07.706187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.718563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.728912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.739187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.741778] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1748678347641249000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137310979}, palf_committed_end_scn_:{val:1748678347641249000, v:0}, last_check_scn_:{val:1748678347641249000, v:0}, max_applied_cb_scn_:{val:1748678347641249000, v:0}}) [2025-05-31 02:59:07.741941] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137310979}, readable_log_tail:{lsn:2137310979}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10041597, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:07.750487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.760744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.773175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.785543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.796043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.807917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.818533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.829814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.832747] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83088][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA7A-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:07.838666] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=105] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3476, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:07.838771] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=92] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=660, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:07.839198] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:07.840194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.844212] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFA9D-0-0] [lt=119] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.677660179138183594e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.676391883357011466e+03) [2025-05-31 02:59:07.852716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.857173] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137311101}], scn_array=[cnt:1, {val:1748678347844241000, v:0}], curr_size=589, accum_size=1300, time_cost=11129) [2025-05-31 02:59:07.857503] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:07.867143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=182][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.867318] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448C-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:07.876687] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27] LogLoopThread round_cost_time(us)(round_cost_time=88) [2025-05-31 02:59:07.877406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.881257] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:07.887640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.897997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.908602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.918950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.932626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.945299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.945412] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=24] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:07.955974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.968089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.977462] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=42] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:07.980207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.991558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:07.993228] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=51] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:08.001821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.015392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.019387] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=13] [APPLY STAT CB APPEND COST TIME](cur_stat_count=57, stat_interval=5000000, avg (count/cost)=159, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:08.019432] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=44] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=57, stat_interval=5000000, avg (count/cost)=6820, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:08.019443] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=57, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:08.019450] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=57, stat_interval=5000000, avg (count/cost)=-960, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:08.019457] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A3886-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=57, stat_interval=5000000, avg (count/cost)=6020, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:08.019704] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665517F55AC-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4103) [2025-05-31 02:59:08.023626] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1748678348023622, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007391, add_bytes: 0 [2025-05-31 02:59:08.026256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.026527] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:08.037518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.050783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.054325] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=167] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:08.054363] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7386, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:08.057683] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:08.061154] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:08.063165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.068145] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=151] current monitor number(seq_=-1) [2025-05-31 02:59:08.068234] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=22] gc stale ls task succ [2025-05-31 02:59:08.069825] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=85] PNIO [ratelimit] time: 1748678348069821, bytes: 7134468360, bw: 0.326671 MB/s, add_ts: 1000566, add_bytes: 342733 [2025-05-31 02:59:08.075902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.082412] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:08.086593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.090094] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=51] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:08.097437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.103731] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=71] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:08.107743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.118781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.132058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=193][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.142284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.155091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.166520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.178453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.187790] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206F-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:08.187831] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A206F-0-0] [lt=32] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=44) [2025-05-31 02:59:08.188676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.192275] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:08.192332] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44351, [delay/us]:prepare:24, schedule:199, submit:756, rt:3253, total:4234) [2025-05-31 02:59:08.192350] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:08.192453] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44355, [delay/us]: prepare:24, schedule:199, submit:756, device:3253, total:4234) [2025-05-31 02:59:08.192471] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:08.192498] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4032, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:319, total:337) [2025-05-31 02:59:08.192525] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:08.192557] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4032, [delay/us]: prepare:0, schedule:0, submit:600000000, device:319, total:337) [2025-05-31 02:59:08.192576] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:08.192590] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:08.192598] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:08.192607] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:08.192615] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:08.192624] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=357kB/s, net_out=292kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:08.192644] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399443][2nd=3016]}) [2025-05-31 02:59:08.200413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.211316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.222061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.222122] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:08.222246] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [25162][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=76] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-05-31 02:59:08.233231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.235409] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC10871E-0006366541D8587C-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678348233964000, start_id=1748678348233964000, end_id=1748678348233964001) [2025-05-31 02:59:08.241044] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=37] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:173833 disk:48K/11:173833 t1m2:48K/11:173833 t1m2g0:48K/11:173833 |q0: 0 |q1: 0 [2025-05-31 02:59:08.246179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.257661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.258923] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=144] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:08.259447] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:08.261618] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:08.269755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.279932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.285447] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:08.292984] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A567F-0-0] [lt=263] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678343633608, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:08.293054] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A567F-0-0] [lt=67] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678343635970, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:08.293081] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81063][TimerWK1_DumpLSLoc][T0][YB42AC1087A6-00063665443A567F-0-0] [lt=22] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678343647176, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678343647183, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:08.294433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.304631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.315093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.317955] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A388F-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525397500124, cache_obj->added_lc()=false, cache_obj->get_object_id()=53665, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:08.325959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.336248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.342115] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=43] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:08.342142] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:08.342196] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F47-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:08.342218] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F47-0-0] [lt=18] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:08.346533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.358953] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.359010] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.359038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.359074] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.359347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.369788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.373388] INFO pn_ratelimit (group.c:653) [81063][TimerWK1_IngressService][T0][Y0-0000000000000000-0-0] [lt=73] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:08.380195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.390895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.396447] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:08.401447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.411733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.422105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.432471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.432842] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:08.446238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:08.446963] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [26238][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:08.447257] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-05-31 02:59:08.453202] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678348045078001, v:0}, new_max_version={val:1748678348045078001, v:0}, last_min_version={val:1748678348045078000, v:0}, last_max_version={val:1748678348045078000, v:0}, total_time_us=6073, delta_us=408120) [2025-05-31 02:59:08.453256] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678348045078001, v:0}, min_version={val:1748678348045078001, v:0}, max_version={val:1748678348045078001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:08.459382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.459734] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:08.464725] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678348464696], gts:1748678348463809000, latest_srr:[mts=1748678348464696]}) [2025-05-31 02:59:08.464765] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:08.470015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.480416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.486238] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:08.490809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.501372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=250][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.512030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.519091] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff896450, session_id:3221738425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678347698812, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1019, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:08.519175] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=71] kill and revert session(conn.sessid_=3221738425, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:59:08.519192] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] sock ref clean, do destroy(*s={this:0x2b08ff896450, session_id:3221738425, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678347698812, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:1019, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:08.519324] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221738425, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:59:08.519342] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221738425, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654C3A388E-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:11, cur_pkt_pos_:17, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:9340386, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:9340387, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:9340388, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:9340388, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:9340388, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:9340388, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:9340388, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:9340388, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:9340388, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:9340389, flag_.flags:30, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[11]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[14]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:16}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[15]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:17}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[16]:{send:obp_compess_header_{com_len:794, com_seq:1}, obp20_header_:{payload_len:766, pkt_seq:1, req_id:9340389, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:18}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:59:08.519438] INFO [SERVER] run (obmp_disconnect.cpp:78) [83079][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=43] free session successfully(sessid=3221738425) [2025-05-31 02:59:08.519459] INFO [SERVER] run (obmp_disconnect.cpp:85) [83079][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=19] mark session id unused(sessid=3221738425) [2025-05-31 02:59:08.519465] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=56][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff9f2bd0, session_id:3221736916, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:153}, err:5, last_decode_time:1748678330342490, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:861, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:08.519496] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221736916, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:59:08.519504] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sock ref clean, do destroy(*s={this:0x2b08ff9f2bd0, session_id:3221736916, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:153}, err:5, last_decode_time:1748678330342490, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:861, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:08.519556] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] unlock session of tenant(conn.sessid_=3221736916, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:59:08.519568] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221736916, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654CCA1C83-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:3991589, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:3991590, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:321, pkt_seq:0, req_id:3991591, flag_.flags:30, ext_flags:0}, obp_mysql_header_{len_:317, rec_:321, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:376, com_seq:1}, obp20_header_:{payload_len:348, pkt_seq:1, req_id:3991591, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:376, com_seq:1}, obp20_header_:{payload_len:348, pkt_seq:1, req_id:3991591, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:4, seq_:6}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:376, com_seq:1}, obp20_header_:{payload_len:348, pkt_seq:1, req_id:3991591, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:7}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:376, com_seq:1}, obp20_header_:{payload_len:348, pkt_seq:1, req_id:3991591, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:2, seq_:9}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:3991591, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:3991591, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:11}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:59:08.520262] INFO [SERVER] run (obmp_disconnect.cpp:78) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=5] free session successfully(sessid=3221736916) [2025-05-31 02:59:08.520301] INFO [SERVER] run (obmp_disconnect.cpp:85) [83078][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=38] mark session id unused(sessid=3221736916) [2025-05-31 02:59:08.522388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.528631] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=220] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:08.534756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.545165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.555579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.563545] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:08.565942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.576158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.586453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.590349] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.590549] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.590612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.591456] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.591497] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.591960] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.592082] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:08.596826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.607179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.617565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.627747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.638192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=177][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.641058] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=81, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:08.641113] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3981, avg_group_log_size=398) [2025-05-31 02:59:08.641135] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3981) [2025-05-31 02:59:08.641151] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:08.647922] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D4-0-0] [lt=34] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:08.650495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=185][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.653419] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137314249}], scn_array=[cnt:1, {val:1748678348463809000, v:0}], curr_size=122, accum_size=3981, time_cost=12187) [2025-05-31 02:59:08.653476] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3981) [2025-05-31 02:59:08.653670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=163, avg_log_gen_to_submit_time=264, avg_log_submit_to_flush_time=6979, avg_log_submit_to_slide_time=8483) [2025-05-31 02:59:08.653775] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=343] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3981, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:08.662191] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:08.662576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.673470] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=33] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:08.673541] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=49] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=965) [2025-05-31 02:59:08.673583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.680750] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC10871E-000636654E089A3C-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525397862919, cache_obj->added_lc()=false, cache_obj->get_object_id()=53666, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:08.686803] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:08.688400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.698762] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.709079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.719325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.729691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.740113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=182][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.746093] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:08.746137] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-05-31 02:59:08.746149] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:08.746165] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:08.746181] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525396926210, flush_io_finished_ret_:0, flush_io_finished_round_:19916, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:08.750848] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.761169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.771373] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.783013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.795369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.806216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.812966] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:59:08.813010] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=32] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52389880, tenant_id=1) [2025-05-31 02:59:08.813036] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:08.816158] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:08.816321] WDIAG read_multi_row (ob_table_access_helper.h:238) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:08.816522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.818248] INFO delete_row (ob_table_access_helper.h:339) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678348816340) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678348816340) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:59:08.821136] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=102] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:08.829049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.839629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.846647] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6049, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:08.846711] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=654, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:08.849876] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:08.850400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.851727] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA7B-0-0] [lt=40][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:08.853435] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAA8-0-0] [lt=21] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.677660179138183594e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.676391883357011466e+03) [2025-05-31 02:59:08.853713] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] get number(ret=0, service_type_=0, range=1, base_id=1748678348853333000, start_id=1748678348853333000, end_id=1748678348853333001) [2025-05-31 02:59:08.861192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.865621] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:08.871116] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448D-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:08.871692] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.881937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.889402] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:08.893955] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=259][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.904282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.908621] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81744][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=31] clear dtl interm result cost(us)(clear_cost=3937, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:59:08.915116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.925401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.935177] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=97] XA scheduler heartbeat task statistics(avg_time=2) [2025-05-31 02:59:08.935797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.941734] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3E-0-0] [lt=42] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678348940930, keepalive_ts:-1}) [2025-05-31 02:59:08.941787] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3E-0-0] [lt=50] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678348940930, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:08.945638] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:08.946064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.946162] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:59:08.956745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.966512] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678348966070000, start_id=1748678348966070000, end_id=1748678348966070001) [2025-05-31 02:59:08.967341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=175][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.978127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:08.990141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.000407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.010808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.021136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.029043] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:09.031476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.031988] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678349031983, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008361, add_bytes: 0 [2025-05-31 02:59:09.041659] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.048285] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=33] ====== check clog disk timer task ====== [2025-05-31 02:59:09.048388] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:09.049438] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC117C-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3433) [2025-05-31 02:59:09.056365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.059985] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:09.060015] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7032, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:09.064560] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:09.066546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.068722] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:09.071529] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:59:09.071597] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=18] gc stale ls task succ [2025-05-31 02:59:09.071739] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=102] PNIO [ratelimit] time: 1748678349071736, bytes: 7134669524, bw: 0.191478 MB/s, add_ts: 1001915, add_bytes: 201164 [2025-05-31 02:59:09.078243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.088586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.089992] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=76] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:09.090337] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=46] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:09.103036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.103921] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=47] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:09.113355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.126078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.138486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.151275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.162492] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.173020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.184991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.195502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.203651] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:09.203712] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44549, [delay/us]:prepare:12, schedule:303, submit:301, rt:4072, total:4690) [2025-05-31 02:59:09.203781] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:09.203911] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44546, [delay/us]: prepare:12, schedule:303, submit:301, device:4072, total:4690) [2025-05-31 02:59:09.203929] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:09.203956] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4049, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:449, total:464) [2025-05-31 02:59:09.203983] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:09.204216] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=225] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4048, [delay/us]: prepare:0, schedule:0, submit:600000000, device:449, total:464) [2025-05-31 02:59:09.204244] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:09.204264] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:09.204272] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:09.204281] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:09.204289] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:09.204298] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=170kB/s, net_out=103kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:09.204319] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399444][2nd=1465]}) [2025-05-31 02:59:09.206031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.216475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.226722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.238442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.246670] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=26] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:40K/9:309216 disk:40K/9:309216 t1m2:40K/9:309216 t1m2g0:40K/9:309216 |q0: 0 |q1: 0 [2025-05-31 02:59:09.251654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.261924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:09.264530] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [26524][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:09.264750] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:09.269534] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=122] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:09.275364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.278881] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0DCA-0-0] [lt=34] handle gts request statistics(total_rt=1670, total_cnt=181, avg_rt=9.175824175824175200e+00, avg_cnt=1.810000000000000142e+01) [2025-05-31 02:59:09.285739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.292413] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:09.293781] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2025-05-31 02:59:09.296659] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.302159] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=52] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:09.307111] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.317907] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.328191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.335217] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83106][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3891-0-0] [lt=62][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525398517385, cache_obj->added_lc()=false, cache_obj->get_object_id()=53667, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:09.338467] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.350844] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.359192] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.359254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.359285] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.359310] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.362104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.373023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=133][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.384705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.394953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.407678] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=29] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:09.409396] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=42] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:09.409741] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=98] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678342188585000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:09.410022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.420363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.429787] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=47] blacklist_loop exec finished(cost_time=25, is_enabled=true, send_cnt=0) [2025-05-31 02:59:09.430970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=250][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.442546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=292][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.453364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.465036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.466211] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678349257515000, v:0}, new_max_version={val:1748678349257515000, v:0}, last_min_version={val:1748678348966070000, v:0}, last_max_version={val:1748678348966070000, v:0}, total_time_us=10693, delta_us=208692) [2025-05-31 02:59:09.466256] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678349257515000, v:0}, min_version={val:1748678349257515000, v:0}, max_version={val:1748678349257515000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:09.466282] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:09.468062] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=41] ts response task statistics(alloc_count=486530, free_count=243264) [2025-05-31 02:59:09.468109] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:09.468215] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:09.468347] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=122] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678348966590], gts:1748678348991221000, latest_srr:[mts=1748678349468130]}) [2025-05-31 02:59:09.471565] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:09.477061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.487747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.494481] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:09.498271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.508528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.519006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.529325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.529654] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=81] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:09.539629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.550037] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.557084] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678349454779000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678349557029}, version_delta=102302) [2025-05-31 02:59:09.562525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.565344] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:09.572838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.585778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=147][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.590817] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.590870] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.590900] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.591289] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.593746] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83082][T1_L0_G0][T1][YB42AC108714-000636671E2D3A70-0-0] [lt=61] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:09.593997] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.594034] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.594338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:09.599351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.609679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.620028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.630338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.638349] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83086][T1_L0_G0][T1][YB42AC1087A8-000636672F8E3F33-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1748678349631346000, start_id=1748678349631346000, end_id=1748678349631346001) [2025-05-31 02:59:09.641083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.657620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=643][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.661455] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:09.661563] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4569, avg_group_log_size=415) [2025-05-31 02:59:09.661587] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4102) [2025-05-31 02:59:09.661624] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:09.666736] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4569) [2025-05-31 02:59:09.670995] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=95, avg_log_gen_to_submit_time=217, avg_log_submit_to_flush_time=5799, avg_log_submit_to_slide_time=6081) [2025-05-31 02:59:09.671481] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4569, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 03:59:09.672641] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [26649][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:09.672647] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=377472, replace_num=15728) [2025-05-31 02:59:09.672679] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:09.672774] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:09.682946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.693242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.697246] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:09.703459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.713694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.724003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.735241] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.737370] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1748678339730664, end_time=1748678349737366, dag_status={}) [2025-05-31 02:59:09.737410] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=35] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737434] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737455] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:09.737502] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:59:09.737511] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:59:09.745117] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83067][T1_L5_G0][T1][YB42AC108714-000636671E7D5D7D-0-0] [lt=40] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=280, Holding Memory(MB)=134, Tenant Remain Memory(MB)=243, Usable Remain Memory(MB)=146, Last Update Limit Timestamp=1748678349744561, Is Updated=true) [2025-05-31 02:59:09.745375] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC108714-000636671E7D5D7D-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1748678349744561000, start_id=1748678349744561000, end_id=1748678349744561001) [2025-05-31 02:59:09.745549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.754503] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83074][T1_L0_G0][T1][YB42AC10871E-0006366543F8A3FA-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1748678349750111000, start_id=1748678349750111000, end_id=1748678349750111001) [2025-05-31 02:59:09.756560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.766808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.777305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.789231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.799986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.813151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.823498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.833792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.845884] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.856709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.861597] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAB7-0-0] [lt=11] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.629660224914550781e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=281, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.768027363835915367e+03) [2025-05-31 02:59:09.865203] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4254, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:09.865284] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=72] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=700, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:09.865378] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=59, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:09.868110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.868936] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=79] BlockGCTimerTask success(ret=0, cost_time_us=18, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:59:09.871112] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA7C-0-0] [lt=21][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:09.872111] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [26238][T1_L0_G1][T1][YB42AC1087A7-00063665517F55C0-0-0] [lt=28] init point thread id with(&point=0x55c0908e1340, idx_=1662, point=[thread id=26238, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=26238) [2025-05-31 02:59:09.874505] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448E-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:09.875400] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:09.878338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.879644] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=35] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19334873088, fast_delete_log_mode=false) [2025-05-31 02:59:09.888733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.897786] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:09.899516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.909956] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:76, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183852855544, recv_hp_rpc_cnt:74896, recv_np_rpc_cnt:878105, recv_lp_rpc_cnt:0, recv_mysql_cnt:47069, recv_task_cnt:2474, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1232535 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9880, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:544077, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1748678349657671}{group_id:10, queue_size:0, recv_req_cnt:103971, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18369, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678341246306}{group_id:2, queue_size:0, recv_req_cnt:150947, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101128, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:107949, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272582, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=684 pcode=0x717:cnt=484 pcode=0x1502:cnt=264 pcode=0x710:cnt=161 pcode=0x523:cnt=99, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:59:09.911049] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1089] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82374330, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:59:09.911655] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=605] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82892323, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:59:09.912433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=768][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.925047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.935726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.945739] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:09.946408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.957155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.961373] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678349750111000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=211247, in_cluster_service=true, cluster_version={val:1748678349631346000, v:0}, min_cluster_version={val:1748678349631346000, v:0}, max_cluster_version={val:1748678349631346000, v:0}, get_cluster_version_err=0, cluster_version_delta=330012, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195341, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=340906600, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:09.967444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.969409] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678349468601], gts:1748678349504701000, latest_srr:[mts=1748678349969382]}) [2025-05-31 02:59:09.977757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.988488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:09.999788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.010098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.020319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.030010] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:10.030739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.040981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.046369] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1748678350046364, bytes: 0, bw: 0.000000 MB/s, add_ts: 1014381, add_bytes: 0 [2025-05-31 02:59:10.052051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.054357] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4954) [2025-05-31 02:59:10.055187] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [26524][T1_L0_G1][T1][YB42AC10871E-0006366549CC1187-0-0] [lt=29] init point thread id with(&point=0x55c0908e5ac0, idx_=1948, point=[thread id=26524, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=26524) [2025-05-31 02:59:10.062474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.066724] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=68] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:10.072075] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=26] current monitor number(seq_=-1) [2025-05-31 02:59:10.072433] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=17] gc stale ls task succ [2025-05-31 02:59:10.072594] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1748678350072590, bytes: 7134829463, bw: 0.152400 MB/s, add_ts: 1000854, add_bytes: 159939 [2025-05-31 02:59:10.072771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.073119] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:10.073147] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6262, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:10.077261] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:10.083745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.091090] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:10.093979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.098567] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:10.103686] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81744][T1_TimerWK3_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=51] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:10.104293] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:10.104424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.114671] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.124885] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.135330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.144451] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81431][pnio1][T0][YB42AC10870A-000636671AB94735-0-0] [lt=6][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:573826844, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678350171122, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1386430, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678350163091, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678350144445, enqueue_timestamp_:0, request_arrival_time_:1748678350144445, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:10.144532] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81431][pnio1][T0][YB42AC10870A-000636671AB94735-0-0] [lt=65][errcode=-5150] deliver rpc request fail(&req=0x2b0959604098, ret=-5150) [2025-05-31 02:59:10.145581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.148199] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83089][T1_L0_G0][T1][YB42AC10870A-000636671578EF75-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678350145257000, start_id=1748678350145257000, end_id=1748678350145257001) [2025-05-31 02:59:10.155860] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.166951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.179140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.189358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.192326] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF6-0-0] [lt=45] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:10.195120] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2070-0-0] [lt=93] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:10.195159] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2070-0-0] [lt=30] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=42) [2025-05-31 02:59:10.199817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.200280] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AF8-0-0] [lt=54] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:10.210753] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.216297] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:10.216434] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=99] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, norm_iops:3, bw:56623, [delay/us]:prepare:40, schedule:590, submit:114, rt:2891, total:3637) [2025-05-31 02:59:10.216454] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:10.216562] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56626, [delay/us]: prepare:40, schedule:590, submit:114, device:2891, total:3637) [2025-05-31 02:59:10.216580] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:10.216608] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4044, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:417, total:432) [2025-05-31 02:59:10.216684] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:10.216724] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4045, [delay/us]: prepare:0, schedule:0, submit:600000000, device:417, total:432) [2025-05-31 02:59:10.216744] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:10.216759] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:10.216767] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:10.216776] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:10.216784] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:10.216793] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=154kB/s, net_out=99kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:10.216814] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399445][2nd=2311]}) [2025-05-31 02:59:10.220976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.231272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.241536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.252324] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=28] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:60K/13:528708 disk:60K/13:528708 t1m2:60K/13:528708 t1m2g0:60K/13:528708 |q0: 0 |q1: 0 [2025-05-31 02:59:10.254615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.264899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.275127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.278591] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:10.287081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.298080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.299412] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:10.308278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.312596] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4221-0-0] [lt=58] get number(ret=0, service_type_=0, range=1, base_id=1748678350311363000, start_id=1748678350311363000, end_id=1748678350311363001) [2025-05-31 02:59:10.318491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.328704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.341367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.345386] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83075][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3893-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525399527555, cache_obj->added_lc()=false, cache_obj->get_object_id()=53668, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.347344] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=28] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:10.347371] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:10.347696] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109B-0-0] [lt=87] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:10.347724] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109B-0-0] [lt=23] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:10.351815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.358932] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.358990] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.359019] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.359046] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.362443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.374478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.385368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.396174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.396246] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83105][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA7F-0-0] [lt=77][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525399578389, cache_obj->added_lc()=false, cache_obj->get_object_id()=53669, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.398063] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:10.398386] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=1995, gc_interval=10000000) [2025-05-31 02:59:10.398424] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=17] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:59:10.398444] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] gc_check_member_list_ cost time(ret=0, time_us=41) [2025-05-31 02:59:10.398457] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:59:10.398532] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=66] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:59:10.398548] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:59:10.398559] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=13) [2025-05-31 02:59:10.398631] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=68] ObSafeDestroyHandler start process [2025-05-31 02:59:10.407427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.417827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.430134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=131][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.440711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.451134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.462647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.470124] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=31][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:10.473401] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=730, get_gts_with_stc_cnt=32, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=31, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:10.473556] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:10.473579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.477386] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678350145257001, v:0}, new_max_version={val:1748678350145257001, v:0}, last_min_version={val:1748678349968205000, v:0}, last_max_version={val:1748678349968205000, v:0}, total_time_us=11908, delta_us=332125) [2025-05-31 02:59:10.477447] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678350145257001, v:0}, min_version={val:1748678350145257001, v:0}, max_version={val:1748678350145257001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:10.479142] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:10.484167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.494452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.500045] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40198144, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:10.504744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.515854] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.527985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=158][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.530622] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:10.534026] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3895-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525399716195, cache_obj->added_lc()=false, cache_obj->get_object_id()=53670, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.535221] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3897-0-0] [lt=44][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:10.537191] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3899-0-0] [lt=17][errcode=-4201] hashtable not init [2025-05-31 02:59:10.537351] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1748678350536792000, start_id=1748678350536792000, end_id=1748678350536792001) [2025-05-31 02:59:10.538566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.543517] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] ====== checkpoint timer task ====== [2025-05-31 02:59:10.544327] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:10.544374] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=46] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:10.544726] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:10.544878] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=145] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:10.544945] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=59] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:10.544960] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678343734669001, v:0}) [2025-05-31 02:59:10.544968] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:10.545031] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=60] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:10.545049] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:10.545070] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:10.545084] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:10.545104] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:10.545126] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:10.545136] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:10.548911] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.559322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.566964] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:10.570142] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.570291] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=32] ObArchiveSender is running(thread_index=0) [2025-05-31 02:59:10.581720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.593506] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.593582] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.593612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.594187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.594481] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D513A-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525399776653, cache_obj->added_lc()=false, cache_obj->get_object_id()=53671, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.595725] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.595763] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.605293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.605467] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:10.606088] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=158] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:10.608746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.609731] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:10.609780] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.609824] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.609907] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:10.612070] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] ls blacklist refresh finish(cost_time=38808) [2025-05-31 02:59:10.612401] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81063][TimerWK1_AutoLSLocRpc][T0][YB42AC1087A6-00063665443A5681-0-0] [lt=109] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:59:10.619184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.630145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.640485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.653058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.661503] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D5-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:10.664547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.664670] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:10.665822] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [25226][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=21] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:10.667335] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137323872}], scn_array=[cnt:1, {val:1748678350536792001, v:0}], curr_size=122, accum_size=5054, time_cost=13635) [2025-05-31 02:59:10.670377] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=80, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:10.670639] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5642, avg_group_log_size=434) [2025-05-31 02:59:10.670798] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=153] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5643) [2025-05-31 02:59:10.670817] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:10.673002] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5642) [2025-05-31 02:59:10.673287] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1190-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=17, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=5383, avg_log_submit_to_slide_time=5773) [2025-05-31 02:59:10.673917] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5642, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:10.675148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.677519] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:10.677670] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=111] get wrs ts(ls_id={id:1}, delta=140199, log_service_ts=1748678350536792, min_tx_service_ts=4611686018427387, timestamp={val:1748678350536792000, v:0}) [2025-05-31 02:59:10.677717] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=56, Other_Error_Cnt=0, Submit_Succ_Cnt=44, last_scn="{val:1748678345706851001, v:0}", last_lsn={lsn:2137303654}, last_gts={val:1748678350536792000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:59:10.679335] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:10.685986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.696809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.703092] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:10.707817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.718554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.729048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.740011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.748658] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:10.748700] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-05-31 02:59:10.748711] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:10.748727] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:10.748742] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525398928585, flush_io_finished_ret_:0, flush_io_finished_round_:19918, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:10.750266] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.760485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.772913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.783249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.791822] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1748678350791673000, start_id=1748678350791673000, end_id=1748678350791673001) [2025-05-31 02:59:10.793765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.800948] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672F2E4726-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678350800214000, start_id=1748678350800214000, end_id=1748678350800214001) [2025-05-31 02:59:10.801182] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672F2E4726-0-0] [lt=57] transaction rpc statistics(total_rt=4489, total_process=17, avg_rt=249) [2025-05-31 02:59:10.804107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.805750] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=6] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-05-31 02:59:10.809520] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=166] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137326454}, readable_log_tail:{lsn:2137326454}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10026122, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:10.809625] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=103] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:59:10.809685] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81740][T1_TimerWK0_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:59:10.814320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.824598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.834788] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:10.834830] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=30] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52389880, tenant_id=1) [2025-05-31 02:59:10.834923] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:10.835041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.845273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.855931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.866205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.873071] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4319, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:10.873479] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=476, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:10.873547] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=55, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:10.877649] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A448F-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:10.879534] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:10.882063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:10.891292] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [27077][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:10.891392] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:10.897504] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA7D-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:10.902348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.903562] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:10.911888] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032403, max_lsn:2137327043, max_scn:{val:1748678350870242000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137327043}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095384003}}, last_submit_lsn:{lsn:2137326454}, last_submit_end_lsn:{lsn:2137327043}, last_submit_log_id:4032403, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137326454}, max_flushed_end_lsn:{lsn:2137327043}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137327043}, last_slide_log_id:4032403, last_slide_scn:{val:1748678350870242000, v:0}, last_slide_lsn:{lsn:2137326454}, last_slide_end_lsn:{lsn:2137327043}, last_slide_log_pid:11, last_slide_log_accum_checksum:3169077372, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032404, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137327043}, readable_log_tail:{lsn:2137327043}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10025533, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:59:10.912256] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=357] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678350873839, last_flushed_end_lsn:{lsn:2137327043}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678350873533, last_flushed_end_lsn:{lsn:2137327043}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678350882459, last_flushed_end_lsn:{lsn:2137327043}}]) [2025-05-31 02:59:10.912283] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032404, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678350873839, last_flushed_end_lsn:{lsn:2137327043}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678350873533, last_flushed_end_lsn:{lsn:2137327043}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678350882459, last_flushed_end_lsn:{lsn:2137327043}}]) [2025-05-31 02:59:10.912550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.923797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.934409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=142][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.944723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.946015] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:10.956656] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.956751] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:10.956776] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] try gc retain ctx [2025-05-31 02:59:10.956784] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-05-31 02:59:10.956828] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678350870242000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678340674232001, v:0}, keep_alive_scn_:{val:1748678345706851001, v:0}, update_ts_:1748678350956505}) [2025-05-31 02:59:10.957658] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=113] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:10.966966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.975774] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:10.977313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.987652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:10.998317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.008811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.020184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.030401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.031793] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=49] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:11.040717] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.047789] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678351047784, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001420, add_bytes: 0 [2025-05-31 02:59:11.048637] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-05-31 02:59:11.048687] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=42] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:11.052758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.052956] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=25] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:59:11.059134] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC1195-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=6531) [2025-05-31 02:59:11.062982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.067542] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:11.073354] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAC9-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653660774230957031e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.721726517622167648e+03) [2025-05-31 02:59:11.074391] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=110] current monitor number(seq_=-1) [2025-05-31 02:59:11.074523] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=22] gc stale ls task succ [2025-05-31 02:59:11.074829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.075187] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678351075184, bytes: 7135018509, bw: 0.179822 MB/s, add_ts: 1002594, add_bytes: 189046 [2025-05-31 02:59:11.076392] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=18, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:11.076418] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5183, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:11.079983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:11.085394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.093931] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=108] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:11.094279] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [23570][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA1-0-0] [lt=108] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678351127197, keepalive_ts:-1}) [2025-05-31 02:59:11.094311] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [23570][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA1-0-0] [lt=29] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678351127197, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:11.095732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.104252] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:11.105121] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:11.108941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.119115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.129285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.142696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.152970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.163454] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.171672] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=115] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:11.171717] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=30] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:11.171736] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=955) [2025-05-31 02:59:11.175031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.186560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.197156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.207450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.217739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.229588] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.230281] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13] === Run print tenant memory usage task === [2025-05-31 02:59:11.230379] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=72] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 121,196,544 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 647,544,832 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,351,229,200 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 52,389,880 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 900,067,328 max_mem_memstore_can_get_now= 261,949,426 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:59:11.230552] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,073,150,976 total_hold= 3,323,985,920 used= 1,578,782,720 freelists_hold= 1,494,368,256 total_maps= 2,018 total_unmaps= 806 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 121,196,544 virtual_memory_used= 4,344,737,792 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,000,341,504 count= 477 pushes= 14,527,649 pops= 14,527,172 maps= 1,823 unmaps= 762 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,946 pops= 22,942 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:59:11.230581] INFO print (ob_malloc_time_monitor.cpp:37) [81064][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:59:11.230746] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:11.230797] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4411, iops:13, norm_iops:3, bw:56531, [delay/us]:prepare:12, schedule:85, submit:111, rt:3217, total:3426) [2025-05-31 02:59:11.230838] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:11.230979] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=122] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4411, avg_iops:13, avg_bw:56526, [delay/us]: prepare:12, schedule:85, submit:111, device:3217, total:3426) [2025-05-31 02:59:11.230994] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:11.231053] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4037, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:720, total:736) [2025-05-31 02:59:11.231082] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:11.231144] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4037, [delay/us]: prepare:0, schedule:0, submit:600000000, device:720, total:736) [2025-05-31 02:59:11.231165] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:11.231180] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:11.231188] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:11.231196] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:11.231203] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:11.231221] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SERVER] net_in=172kB/s, net_out=157kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:11.231240] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399446][2nd=2593]}) [2025-05-31 02:59:11.240060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.250601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.264085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.278122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.280181] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:11.281192] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:56K/13:149413 disk:56K/13:149413 t1m2:56K/13:149413 t1m2g0:56K/13:149413 |q0: 0 |q1: 0 [2025-05-31 02:59:11.281242] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC10871E-000636654DFA0DD6-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1748678351279264000, start_id=1748678351279264000, end_id=1748678351279264001) [2025-05-31 02:59:11.288364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.298654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.305221] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:11.306544] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2025-05-31 02:59:11.310207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.320631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.330944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.341347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=161][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.351739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.354713] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A389B-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525400536882, cache_obj->added_lc()=false, cache_obj->get_object_id()=53672, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:11.359030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.359077] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.359105] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.359130] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.362471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.372686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.373562] INFO pn_ratelimit (group.c:653) [81065][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=29] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:11.382956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.393234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.403542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.413905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.424171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.434423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.444643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.449082] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:11.452976] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=115] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:11.454970] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=50] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:11.455112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.456799] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=29] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:11.456820] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=20] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:11.458532] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:11.458599] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=18] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:11.458613] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=11] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:11.458638] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:11.459222] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=578] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:11.459261] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=29] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:11.459273] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:11.459295] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:11.459316] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:11.460834] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:11.460937] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=99] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:11.460948] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=9] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:11.460984] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=6] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:11.460998] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:11.461007] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:11.461019] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:11.461035] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDD-0-0] [lt=14] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:11.465711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.476662] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=216] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678351476639], gts:1748678351475805000, latest_srr:[mts=1748678351476639]}) [2025-05-31 02:59:11.476715] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:11.477551] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71] no need to update min/max version(new_version={val:1748678350975369000, v:0}, min_version={val:1748678350975369000, v:0}, max_version={val:1748678350975369000, v:0}) [2025-05-31 02:59:11.478033] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.482070] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:11.488400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.489352] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [23567][T1_L0_G19][T1][YB42AC10871E-00063665511850D9-0-0] [lt=33] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:11.489388] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [23567][T1_L0_G19][T1][YB42AC10871E-00063665511850D9-0-0] [lt=26] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 03:59:11.494189] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [27289][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:11.494205] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:59:11.504453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.507439] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=118] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:11.514749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.525141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=175][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.532130] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-05-31 02:59:11.533805] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A389D-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525400715974, cache_obj->added_lc()=false, cache_obj->get_object_id()=53673, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:11.535672] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.547064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.557611] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.567768] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:11.567895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.578127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.584447] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678351072600001, v:0}, new_max_version={val:1748678351072600001, v:0}, last_min_version={val:1748678350975369000, v:0}, last_max_version={val:1748678350975369000, v:0}, total_time_us=6692, delta_us=511843) [2025-05-31 02:59:11.584507] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678351072600001, v:0}, min_version={val:1748678351072600001, v:0}, max_version={val:1748678351072600001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:11.588627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.590694] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.591445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.591638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=77] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.591939] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.592404] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.592469] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.592499] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:11.598958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.611346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.622052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.635935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.648975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.659407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.672033] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.674602] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81744][T1_TimerWK3_Flush][T1][Y0-0000000000000000-0-0] [lt=102] ====== traversal_flush timer task ====== [2025-05-31 02:59:11.675061] INFO [MDS] flush (mds_table_mgr.cpp:184) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60D0-0-0] [lt=22] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678351579452000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678360364653000, latest_log_ts:{val:1748678343734669001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:11.675242] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81744][T1_TimerWK3_Flush][T1][YB42AC1087A6-00063665431A60D0-0-0] [lt=172] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:11.682227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.683747] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:11.692440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.702603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.709169] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:11.713325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.724133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.735982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.740288] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=91] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:11.742275] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81742][T1_TimerWK1_ReqMemEvict][T1][YB42AC1087A6-0006366543FA5F63-0-0] [lt=26] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=33) [2025-05-31 02:59:11.742616] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=64] get number(ret=0, service_type_=0, range=1, base_id=1748678351741937000, start_id=1748678351741937000, end_id=1748678351741937001) [2025-05-31 02:59:11.742772] INFO [LIB] stat (utility.h:1211) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=95, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:11.742802] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5377, avg_group_log_size=448) [2025-05-31 02:59:11.742821] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5417) [2025-05-31 02:59:11.742912] INFO [LIB] stat (utility.h:1211) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:11.744967] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=57085699, this={dio_aligned_buf:{buf_write_offset:295, buf_padding_size:3801, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:390335, truncate_used_ts:132278}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=8192, aligned_block_offset=57081856, buf_len=548, total_write_size=53633, total_write_size_after_dio=565248, ob_pwrite_used_ts=539001, count=124, each_pwrite_cost=4346) [2025-05-31 02:59:11.745012] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5377) [2025-05-31 02:59:11.746354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.748752] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [26524][T1_L0_G1][T1][YB42AC1087A7-00063665517F55D6-0-0] [lt=92] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=122, avg_log_submit_to_flush_time=3702, avg_log_submit_to_slide_time=4104) [2025-05-31 02:59:11.749082] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5082, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:11.749456] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:11.749505] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-05-31 02:59:11.749557] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=49] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:11.749575] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:11.749590] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525399930957, flush_io_finished_ret_:0, flush_io_finished_round_:19919, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:11.756610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.766963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.779005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.782816] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:11.782955] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=131] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:11.782973] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=16] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:11.782994] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:11.785708] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:11.786540] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=827] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:11.789239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.801409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.811636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.822066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.832193] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3E8-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525401014362, cache_obj->added_lc()=false, cache_obj->get_object_id()=53674, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:11.832596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.844460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.854682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.866553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.877218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.880439] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:11.880454] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4490-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:11.884742] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=393200, replace_num=15728) [2025-05-31 02:59:11.884881] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:11.887457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.887548] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:11.890919] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [25572][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=33] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:11.899405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.910070] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:11.910175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.917292] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA7E-0-0] [lt=24][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:11.920398] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.931749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.942114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.946162] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=17] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:11.952770] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.963214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.968330] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3348, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:11.968385] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=525, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:11.968465] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=45, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:11.973740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.980668] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] no need to update min/max version(new_version={val:1748678351475805000, v:0}, min_version={val:1748678351475805000, v:0}, max_version={val:1748678351475805000, v:0}) [2025-05-31 02:59:11.984205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:11.994586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.009793] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=207][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.020285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.030624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.032553] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=140] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:12.041031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.045785] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:59:12.046070] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=25] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:59:12.051521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.054131] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=52] PNIO [ratelimit] time: 1748678352054126, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006342, add_bytes: 0 [2025-05-31 02:59:12.061901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.065114] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3514) [2025-05-31 02:59:12.070019] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:12.072342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.076516] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=45] current monitor number(seq_=-1) [2025-05-31 02:59:12.076585] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1748678352076581, bytes: 7135218780, bw: 0.190727 MB/s, add_ts: 1001397, add_bytes: 200271 [2025-05-31 02:59:12.076629] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=39] gc stale ls task succ [2025-05-31 02:59:12.083263] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAD5-0-0] [lt=32] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653812170028686523e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.721437536705674120e+03) [2025-05-31 02:59:12.086125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.086776] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:12.089529] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=115] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:12.089560] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5160, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:12.094236] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=107] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:12.096372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.105291] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:12.107998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.112008] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:12.118794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=427][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.129318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=208][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.147183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=248][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.157560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.167754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.178143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.188497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.200966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.202544] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2071-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:12.202625] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2071-0-0] [lt=69] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=84) [2025-05-31 02:59:12.208117] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:48) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=42] send to rs all task status succeed [2025-05-31 02:59:12.211489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.224149] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.234770] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.245122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=148][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.248726] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:12.248823] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52305, [delay/us]:prepare:12, schedule:572, submit:112, rt:2379, total:3077) [2025-05-31 02:59:12.248842] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:12.248982] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=123] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52306, [delay/us]: prepare:12, schedule:572, submit:112, device:2379, total:3077) [2025-05-31 02:59:12.249005] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:12.249098] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=78] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4023, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2934, total:2950) [2025-05-31 02:59:12.249125] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:12.249158] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4023, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2934, total:2950) [2025-05-31 02:59:12.249177] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:12.249191] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:12.249199] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:12.249207] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:12.249216] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:12.249225] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=210kB/s, net_out=127kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:12.249245] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399447][2nd=2875]}) [2025-05-31 02:59:12.255547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.269484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.282600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.288874] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=84] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:12.293818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.307017] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33CF-0-0] [lt=35][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525401489185, cache_obj->added_lc()=false, cache_obj->get_object_id()=53675, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:12.307294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=149][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.312554] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=130] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:12.322043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.334833] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.345346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.350650] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=41] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:12.350676] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:12.351277] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F48-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:12.351310] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F48-0-0] [lt=29] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:12.355797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.358505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.358551] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.358579] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.358603] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.362164] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A389F-0-0] [lt=97][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525401544335, cache_obj->added_lc()=false, cache_obj->get_object_id()=53676, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:12.366101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.376483] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.391710] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.403717] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.416008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.422799] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:12.422859] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=33] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:12.424569] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=129] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:12.424704] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=50] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:12.424740] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=33] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678352212516000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:59:12.426449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.428499] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:465870 disk:52K/12:465870 t1m2:52K/12:465870 t1m2g0:52K/12:465870 |q0: 0 |q1: 0 [2025-05-31 02:59:12.437306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.447707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.458613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.469073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.479648] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14] ts response task statistics(alloc_count=486602, free_count=243300) [2025-05-31 02:59:12.479710] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:12.479769] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678351978332], gts:1748678352014903000, latest_srr:[mts=1748678352479730]}) [2025-05-31 02:59:12.480950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=447][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.491198] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:12.491537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.491648] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:59:12.491745] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [27289][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:59:12.501891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.512405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.515243] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=59] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:12.524574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.533124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:12.534819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.545123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.557068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.567567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.570202] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=67] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:12.579637] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.587965] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83103][T1_L0_G0][T1][YB42AC108714-000636671E2D3A71-0-0] [lt=53] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:12.590541] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.590588] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.590612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.591829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.592281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.592561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.592652] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.592769] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=81] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:12.594133] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A5685-0-0] [lt=103] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:12.603448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=154][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.613819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.624229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.637519] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.647827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.658391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.669354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.669722] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D6-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:12.681424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.691178] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [26649][T1_L0_G1][T1][YB42AC10871E-0006366549CC11A7-0-0] [lt=33] init point thread id with(&point=0x55c0908e7a00, idx_=2073, point=[thread id=26649, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=26649) [2025-05-31 02:59:12.691767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.693992] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [27077][T1_L0_G1][T1][YB42AC1087A7-00063665517F55E1-0-0] [lt=11109] init point thread id with(&point=0x55c0908ee500, idx_=2501, point=[thread id=27077, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=27077) [2025-05-31 02:59:12.694532] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:12.694754] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678352280778001, v:0}, new_max_version={val:1748678352280778001, v:0}, last_min_version={val:1748678352219730001, v:0}, last_max_version={val:1748678352219730001, v:0}, total_time_us=9356, delta_us=413973) [2025-05-31 02:59:12.694789] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678352280778001, v:0}, min_version={val:1748678352280778001, v:0}, max_version={val:1748678352280778001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:12.702261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=168][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.714272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.716838] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:12.724590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.737598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.751294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.751502] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:12.751536] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-31 02:59:12.751547] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:12.751562] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:12.751577] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525400931803, flush_io_finished_ret_:0, flush_io_finished_round_:19920, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:12.759257] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:12.764103] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [23567][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AF-0-0] [lt=28] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:12.764155] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [23567][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1AF-0-0] [lt=41] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:12.764396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.775220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.785617] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:1748678352684217000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137334724}, palf_committed_end_scn_:{val:1748678352684217000, v:0}, last_check_scn_:{val:1748678352684217000, v:0}, max_applied_cb_scn_:{val:1748678352684217000, v:0}}) [2025-05-31 02:59:12.785658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.785691] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137334724}, readable_log_tail:{lsn:2137334724}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:10017852, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:12.795983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.808575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.819546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.825037] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [83067][T1_L5_G0][T1][YB42AC10870A-000636672468BDE7-0-0] [lt=32] need check schema(new_schema_version=1748674258961328, cached_tenant_schema_version=1748674258961328, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT start_service_time FROM __all_server WHERE svr_ip = ? AND svr_port = ?) [2025-05-31 02:59:12.829961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.842187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.852861] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.853724] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-05-31 02:59:12.853757] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=23] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52389880, tenant_id=1) [2025-05-31 02:59:12.853781] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:12.865323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.870614] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=113] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=96, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:12.870673] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4887, avg_group_log_size=407) [2025-05-31 02:59:12.870696] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5313) [2025-05-31 02:59:12.870712] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:12.874220] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4887) [2025-05-31 02:59:12.874777] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [22344][T1_L0_G1][T1][YB42AC10871E-0006366549CC11A8-0-0] [lt=31] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=131, avg_log_submit_to_flush_time=4328, avg_log_submit_to_slide_time=4981) [2025-05-31 02:59:12.874956] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5182, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:12.875881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.882664] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4491-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:12.886449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=401][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.888516] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=88] get number(ret=0, service_type_=0, range=1, base_id=1748678352887553000, start_id=1748678352887553000, end_id=1748678352887553001) [2025-05-31 02:59:12.893787] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:12.894719] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:12.896877] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.896942] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:12.897655] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [26649][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=98] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:12.907121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.917365] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:12.919916] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.926110] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-05-31 02:59:12.930510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.946233] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83080][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA7F-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:12.946470] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:12.946500] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:12.947001] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] update_check_range(min_check_version=1984, max_check_version=1985, global_version=1986) [2025-05-31 02:59:12.947730] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:12.958019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.960514] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=51] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:12.969227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:12.970560] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3F-0-0] [lt=41] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678352969653, keepalive_ts:-1}) [2025-05-31 02:59:12.970692] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD3F-0-0] [lt=129] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678352969653, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:12.975682] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 154 282 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 51 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 16 1 2 0 1 18 6 3 0 1 20 2 1 0 500 0 89 254 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 1 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 28800 [2025-05-31 02:59:12.975752] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=42] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 900,067,328 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 457,547,776 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 106,954,752 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 12,603,392 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:12.976244] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 457,547,776 used= 347,136,928 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 31 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 22,478,848 used= 22,257,216 count= 33 avg_used= 674,461 block_cnt= 33 chunk_cnt= 21 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,200,768 used= 6,324,224 count= 107 avg_used= 59,104 block_cnt= 107 chunk_cnt= 28 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,850,816 used= 2,408,160 count= 87 avg_used= 27,680 block_cnt= 87 chunk_cnt= 25 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 999,424 used= 490,928 count= 61 avg_used= 8,048 block_cnt= 61 chunk_cnt= 22 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 441,984 used= 425,640 count= 9 avg_used= 47,293 block_cnt= 9 chunk_cnt= 7 mod=CommSysVarFac [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 412,032 used= 395,328 count= 87 avg_used= 4,544 block_cnt= 87 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 253,952 used= 245,760 count= 1 avg_used= 245,760 block_cnt= 1 chunk_cnt= 1 mod=SqlExecutor [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 213,056 used= 179,672 count= 15 avg_used= 11,978 block_cnt= 15 chunk_cnt= 3 mod=SqcHandlerParam [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 100,208 used= 26,000 count= 368 avg_used= 70 block_cnt= 99 chunk_cnt= 25 mod=Coro [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 56,160 used= 43,680 count= 65 avg_used= 672 block_cnt= 60 chunk_cnt= 23 mod=[T]ObWarningBuf [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=SqlSession [MEMORY] hold= 47,360 used= 33,152 count= 74 avg_used= 448 block_cnt= 44 chunk_cnt= 11 mod=OMT_Worker [MEMORY] hold= 45,104 used= 4,800 count= 200 avg_used= 24 block_cnt= 76 chunk_cnt= 28 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=SqlSessiVarMap [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PlanVaIdx [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,968 used= 7,776 count= 1 avg_used= 7,776 block_cnt= 1 chunk_cnt= 1 mod=HashNodeDmChe [MEMORY] hold= 7,968 used= 7,776 count= 1 avg_used= 7,776 block_cnt= 1 chunk_cnt= 1 mod=HashNodeDmId [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,664 used= 6,848 count= 4 avg_used= 1,712 block_cnt= 3 chunk_cnt= 3 mod=DetectManager [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 4,032 used= 2,688 count= 7 avg_used= 384 block_cnt= 7 chunk_cnt= 5 mod=TableLockNode [MEMORY] hold= 4,032 used= 3,840 count= 1 avg_used= 3,840 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlChan [MEMORY] hold= 4,032 used= 3,840 count= 1 avg_used= 3,840 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlTxChan [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=PackStateMap [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=SequenceMap [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=ContextsMap [MEMORY] hold= 1,872 used= 1,272 count= 3 avg_used= 424 block_cnt= 3 chunk_cnt= 3 mod=SequenceIdMap [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,408 used= 1,024 count= 2 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 608 used= 224 count= 2 avg_used= 112 block_cnt= 2 chunk_cnt= 2 mod=Coordinator [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 224 used= 24 count= 1 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=InterrCheckNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 347,136,928 used= 339,022,131 count= 32,641 avg_used= 10,386 mod=SUMMARY [2025-05-31 02:59:12.976315] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=56] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:59:12.976335] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:59:12.976356] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,574,720 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,511,808 used= 1,476,096 count= 186 avg_used= 7,936 block_cnt= 186 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,360,000 used= 1,240,000 count= 625 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,574,720 used= 35,241,449 count= 3,357 avg_used= 10,497 mod=SUMMARY [2025-05-31 02:59:12.976375] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:59:12.976388] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 106,954,752 used= 104,767,488 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 104,767,488 used= 104,572,608 count= 203 avg_used= 515,136 block_cnt= 203 chunk_cnt= 51 mod=CoStack [MEMORY] hold= 104,767,488 used= 104,572,608 count= 203 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:59:12.976408] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:59:12.976424] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:59:12.976437] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:12.976450] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:12.976462] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:59:12.976477] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 40,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 24,576 used= 16,256 count= 1 avg_used= 16,256 block_cnt= 1 chunk_cnt= 1 mod=OB_DTL_SEND [MEMORY] hold= 16,384 used= 8,835 count= 1 avg_used= 8,835 block_cnt= 1 chunk_cnt= 1 mod=[L]OB_PX_FAST_I [MEMORY] hold= 40,960 used= 25,091 count= 2 avg_used= 12,545 mod=SUMMARY [2025-05-31 02:59:12.976496] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 12,603,392 used= 9,018,432 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,827,776 used= 5,690,112 count= 717 avg_used= 7,936 block_cnt= 717 chunk_cnt= 4 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 9,018,432 used= 8,851,776 count= 721 avg_used= 12,277 mod=SUMMARY [2025-05-31 02:59:12.976527] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:12.976565] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 647,544,832 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:12.976995] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 182,490,000 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-05-31 02:59:13.037232] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3236, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:13.037273] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=738, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:13.037468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.038057] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=60, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:13.038151] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=23] [APPLY STAT CB APPEND COST TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=158, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:13.038170] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=19] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=6238, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:13.038178] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:13.038185] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=6] [APPLY STAT CB EXECUTE TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=-1030, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:13.038192] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=50, stat_interval=5000000, avg (count/cost)=5366, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:13.047720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.050240] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=43] ====== check clog disk timer task ====== [2025-05-31 02:59:13.050336] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:13.059357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=465][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.061707] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1748678353061702, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007576, add_bytes: 0 [2025-05-31 02:59:13.069589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.070405] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:13.076751] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=10] gc stale ls task succ [2025-05-31 02:59:13.079825] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.082208] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1748678353082203, bytes: 7135398060, bw: 0.170019 MB/s, add_ts: 1005622, add_bytes: 179280 [2025-05-31 02:59:13.091999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.094603] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:13.095354] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:13.096257] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5312) [2025-05-31 02:59:13.100242] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665517F55E5-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:13.100271] INFO [LIB] stat (utility.h:1211) [22344][T1_L0_G1][T1][YB42AC1087A7-00063665517F55E5-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5167, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:13.102480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.105381] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=33] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:13.113297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.118492] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:13.123899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.130565] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AFA-0-0] [lt=42] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:13.136652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.150010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.160658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.165414] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC1087A7-00063665469F7BDB-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678353162094000, start_id=1748678353162094000, end_id=1748678353162094001) [2025-05-31 02:59:13.172949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=484][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.186007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.196150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.206657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=209][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.217596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.228065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.239262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.250016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.260289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.264639] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:13.265043] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, norm_iops:2, bw:40320, [delay/us]:prepare:13, schedule:230, submit:68, rt:2021, total:2334) [2025-05-31 02:59:13.265070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:13.265131] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40309, [delay/us]: prepare:13, schedule:230, submit:68, device:2021, total:2334) [2025-05-31 02:59:13.265145] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:13.265172] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4031, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:284, total:297) [2025-05-31 02:59:13.265197] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:13.265225] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4031, [delay/us]: prepare:0, schedule:0, submit:600000000, device:284, total:297) [2025-05-31 02:59:13.265244] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:13.265257] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:13.265266] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:13.265274] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:13.265283] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:13.265292] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=174kB/s, net_out=98kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:13.265313] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399448][2nd=3029]}) [2025-05-31 02:59:13.275191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.286414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.290091] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] no need to update min/max version(new_version={val:1748678352887553000, v:0}, min_version={val:1748678352887553000, v:0}, max_version={val:1748678352887553000, v:0}) [2025-05-31 02:59:13.297491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.299058] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:13.309084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.320560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.322093] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:13.323556] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=94] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2025-05-31 02:59:13.334088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.349687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.358620] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.358688] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.359681] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.359758] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.361253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.373446] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38A1-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525402555614, cache_obj->added_lc()=false, cache_obj->get_object_id()=53677, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:13.373676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.385593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.392759] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAE7-0-0] [lt=31] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.677962923049926758e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.675826056474944380e+03) [2025-05-31 02:59:13.395355] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:244350 disk:44K/10:244350 t1m2:44K/10:244350 t1m2g0:44K/10:244350 |q0: 0 |q1: 0 [2025-05-31 02:59:13.395979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.406267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.416642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.427987] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.438171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.448453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.450176] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B35-0-0] [lt=33] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.459122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.470726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.481646] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.483138] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=133] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:13.493772] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.499214] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:13.504412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.514743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.526980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=123][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.527846] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=355] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:13.536158] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=64] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-05-31 02:59:13.537359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.547600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.558353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.571249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.572991] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:13.582062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.595271] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B36-0-0] [lt=24] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.597510] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.597556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.597584] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.598183] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=212] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.598219] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.598213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.598245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.598278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=352] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:13.608909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.609159] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:13.609194] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:13.609325] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:13.609350] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:13.613080] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6] start to print blacklist info [2025-05-31 02:59:13.613546] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=39] ls blacklist refresh finish(cost_time=40114) [2025-05-31 02:59:13.619331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.626941] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B38-0-0] [lt=52] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.631699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.642134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.654502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.664798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.674082] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81744][T1_TimerWK3_TabletShell][T1][Y0-0000000000000000-0-0] [lt=12] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:13.674120] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81744][T1_TimerWK3_TabletShell][T1][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=996) [2025-05-31 02:59:13.677205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=128][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.687958] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B39-0-0] [lt=32] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.687976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.698227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.702709] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:13.709469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.710443] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A40-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525402892612, cache_obj->added_lc()=false, cache_obj->get_object_id()=53678, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:13.720000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.729729] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:13.730423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.740646] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.751222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.751958] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:13.751998] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-05-31 02:59:13.752009] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:13.752025] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:13.752040] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525401933791, flush_io_finished_ret_:0, flush_io_finished_round_:19921, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:13.756066] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=163] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:13.761982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.763081] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1ECF-0-0] [lt=30] submit log success(service_type_=0, last_id=1748678370364653000, limited_id=1748678370364653000) [2025-05-31 02:59:13.763109] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1ECF-0-0] [lt=28] submit log with lock success(service_type_=0, last_id=1748678370364653000, limited_id=1748678370364653000) [2025-05-31 02:59:13.768589] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1748678370364653000, latest_scn={val:1748678353762182001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:59:13.768668] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=78] handle submit callback(service_type_=0, success=true, used_ts=5589, log_ts={val:1748678353762182001, v:0}, limited_id=1748678370364653000) [2025-05-31 02:59:13.768684] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:59:13.768694] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A389A-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:59:13.772359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.774719] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B3A-0-0] [lt=31] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.785007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.795541] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=116] no need to update min/max version(new_version={val:1748678353090653002, v:0}, min_version={val:1748678353090653002, v:0}, max_version={val:1748678353090653002, v:0}) [2025-05-31 02:59:13.795709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.805928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.819421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.829846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.840122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.852757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.863085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.875376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.878654] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=90, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:13.878711] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3519, avg_group_log_size=351) [2025-05-31 02:59:13.878735] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3519) [2025-05-31 02:59:13.878751] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:13.884352] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3519) [2025-05-31 02:59:13.886700] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4492-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:13.886994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.887164] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=124, avg_log_submit_to_flush_time=6659, avg_log_submit_to_slide_time=7934) [2025-05-31 02:59:13.887400] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3519, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:13.897235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.897295] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:13.897476] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B3B-0-0] [lt=50] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.902288] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [22344][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=8] worker thread exit(tenant_->id()=1, workers_.get_size()=6) [2025-05-31 02:59:13.904794] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:13.906032] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678353587685000, v:0}, new_max_version={val:1748678353587685000, v:0}, last_min_version={val:1748678353090653002, v:0}, last_max_version={val:1748678353090653002, v:0}, total_time_us=8496, delta_us=318343) [2025-05-31 02:59:13.906072] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678353587685000, v:0}, min_version={val:1748678353587685000, v:0}, max_version={val:1748678353587685000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:13.907580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.917914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.928172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.930686] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:13.934196] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B3C-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:13.939674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.949024] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=127] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:13.949996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.961472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.971658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.975552] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83106][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA80-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:13.976258] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=39] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:13.981947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.985034] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=63] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:13.994431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:13.997740] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:14.004889] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.015181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.027560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.037658] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:14.038577] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.049209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.060818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.070079] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678354070071, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008369, add_bytes: 0 [2025-05-31 02:59:14.071330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.075709] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:14.076993] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2025-05-31 02:59:14.077126] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=29] gc stale ls task succ [2025-05-31 02:59:14.081740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.090279] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1748678354090273, bytes: 7135574752, bw: 0.167158 MB/s, add_ts: 1008070, add_bytes: 176692 [2025-05-31 02:59:14.091987] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.094761] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:14.102448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.104293] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5420, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:14.104336] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=843, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:14.104417] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=69] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=2972) [2025-05-31 02:59:14.104559] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=137] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=81, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:14.105996] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=408928, replace_num=15728) [2025-05-31 02:59:14.106074] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=75] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:14.106116] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=60] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:14.108221] INFO [LIB] stat (utility.h:1211) [25121][T1_L0_G1][T1][YB42AC10871E-0006366549CC11B4-0-0] [lt=39] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:14.108259] INFO [LIB] stat (utility.h:1211) [25121][T1_L0_G1][T1][YB42AC10871E-0006366549CC11B4-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=8239, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:14.112695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.123051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.131473] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=63] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:14.133265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.145650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=128][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.156391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.166547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.170338] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=21] finish to resize log external storage handler(current_ts=1748678354170332, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:59:14.170419] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=78] snapshot_traffic_info once [2025-05-31 02:59:14.178216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.189517] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.200829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.209512] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2072-0-0] [lt=34] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:14.209569] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2072-0-0] [lt=46] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=59) [2025-05-31 02:59:14.212014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.222501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.232791] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.244421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=133][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.254674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.264917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.273095] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:14.273179] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, norm_iops:2, bw:40615, [delay/us]:prepare:23, schedule:765, submit:213, rt:4126, total:5129) [2025-05-31 02:59:14.273199] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:14.273283] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40628, [delay/us]: prepare:23, schedule:765, submit:213, device:4126, total:5129) [2025-05-31 02:59:14.273300] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:14.273326] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4062, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:547, total:560) [2025-05-31 02:59:14.273351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:14.273412] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4062, [delay/us]: prepare:0, schedule:0, submit:600000000, device:547, total:560) [2025-05-31 02:59:14.273433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:14.273447] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:14.273455] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:14.273462] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:14.273495] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:14.273506] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=164kB/s, net_out=136kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:14.273527] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399449][2nd=1840]}) [2025-05-31 02:59:14.277956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.288273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.300743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.304965] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=72] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:14.306665] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:14.311163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.321250] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4239-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1748678354320537000, start_id=1748678354320537000, end_id=1748678354320537001) [2025-05-31 02:59:14.321596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.333265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.334000] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:14.343636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.354042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.354246] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=13] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:14.354294] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=47] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:14.354368] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F49-0-0] [lt=13] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:14.354416] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F49-0-0] [lt=43] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:14.359007] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.359099] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.359188] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=189] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.359509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.364283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.374436] INFO pn_ratelimit (group.c:653) [81063][TimerWK1_IngressService][T0][Y0-0000000000000000-0-0] [lt=45] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:14.375078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.381687] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38A3-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525403563854, cache_obj->added_lc()=false, cache_obj->get_object_id()=53679, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:14.385623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.396008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.406263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.416583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.424367] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:14.426964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.433238] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=58] blacklist_loop exec finished(cost_time=21, is_enabled=true, send_cnt=0) [2025-05-31 02:59:14.437250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.448220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.459371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.470058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.480996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.485968] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678354485933], gts:1748678354485592000, latest_srr:[mts=1748678354485933]}) [2025-05-31 02:59:14.486104] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=130] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:14.486283] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:14.492216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.503180] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.506016] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFAF6-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.653963518142700195e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=282, Throttle Trigger(MB)=169, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.721148685141645274e+03) [2025-05-31 02:59:14.506633] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=43] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:627668 disk:44K/10:627668 t1m2:44K/10:627668 t1m2g0:44K/10:627668 |q0: 0 |q1: 0 [2025-05-31 02:59:14.507378] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=275] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:14.511553] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B3D-0-0] [lt=92] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:14.514049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:14.515242] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [28331][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 03:59:14.515296] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665517F55F2-0-0] [lt=23] init point thread id with(&point=0x55c090901e80, idx_=3755, point=[thread id=28331, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=28331) [2025-05-31 02:59:14.515445] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1, group_id_=1, token=6) [2025-05-31 02:59:14.527486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.535394] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:14.537907] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.538268] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:14.550294] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38A5-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525403732460, cache_obj->added_lc()=false, cache_obj->get_object_id()=53680, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:14.552600] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B3E-0-0] [lt=69] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:14.553017] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38A7-0-0] [lt=53][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:14.554325] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38A9-0-0] [lt=79][errcode=-4201] hashtable not init [2025-05-31 02:59:14.554473] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:670) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=42] failed to get result(ret=-4008) [2025-05-31 02:59:14.554665] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=73] get number(ret=0, service_type_=0, range=1, base_id=1748678354554142000, start_id=1748678354554142000, end_id=1748678354554142001) [2025-05-31 02:59:14.554659] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:515) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=40] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748678246784954, start_date_:1748169746138009, last_date_:1748678246141150, this_date_:0, next_date_:1748679146138009, end_date_:64060560000000000, total_:239131641, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748613666964441, start_date_:1748613600000000, last_date_:1748613600006788, this_date_:0, next_date_:1749218400000000, end_date_:64060560000000000, total_:66957653, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748314024043185, start_date_:1748268000000000, last_date_:0, this_date_:0, next_date_:1748872800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748600847610427, start_date_:1748255246137533, last_date_:1748600846140058, this_date_:0, next_date_:1748687246137533, end_date_:64060560000000000, total_:3757845, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748168846136670, start_date_:1748700000000000, last_date_:0, this_date_:0, next_date_:1748700000000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748624400066065, start_date_:1748192400000000, last_date_:1748624400003705, this_date_:0, next_date_:1748710800000000, end_date_:64060560000000000, total_:347108, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:4, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748181773502269, start_date_:1748181600000000, last_date_:1748181600002317, this_date_:0, next_date_:1748786400000000, end_date_:64060560000000000, total_:173499952, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748527281005724, start_date_:1748527200000000, last_date_:1748527200011176, this_date_:0, next_date_:1749132000000000, end_date_:64060560000000000, total_:80994548, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748437312134789, start_date_:1748354400000000, last_date_:0, this_date_:0, next_date_:1748959200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748440876223997, start_date_:1748440800000000, last_date_:1748440800003333, this_date_:0, next_date_:1749045600000000, end_date_:64060560000000000, total_:76220664, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}]) [2025-05-31 02:59:14.554791] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:504) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=130] check all tenants(ret=0) [2025-05-31 02:59:14.554968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.566911] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.576374] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=78] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:14.577192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.590010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=231][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.590374] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.590662] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.590719] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.590942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.590985] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.591013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.592681] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:14.600477] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.601610] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678354505690000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678354601549}, version_delta=95917) [2025-05-31 02:59:14.610753] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.621810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.633235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.643763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.655948] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83103][T1_L0_G0][T1][YB42AC1087A8-000636672F7E3EC6-0-0] [lt=59] get number(ret=0, service_type_=0, range=1, base_id=1748678354653671000, start_id=1748678354653671000, end_id=1748678354653671001) [2025-05-31 02:59:14.657834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.669965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.678621] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D7-0-0] [lt=37] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:14.681725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.692615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.706302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.708334] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:14.716831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.728190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.737314] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:14.738644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.749065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.759587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.760814] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83085][T1_L0_G0][T1][YB42AC108714-000636671ECD5CBE-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1748678354760104000, start_id=1748678354760104000, end_id=1748678354760104001) [2025-05-31 02:59:14.770118] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83080][T1_L0_G0][T1][YB42AC10871E-000636654378A52F-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1748678354766457000, start_id=1748678354766457000, end_id=1748678354766457001) [2025-05-31 02:59:14.771139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.781684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.792809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.803323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.814054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.826321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.837440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.849749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.862104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.873459] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:14.873501] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=33] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52389880, tenant_id=1) [2025-05-31 02:59:14.873526] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:14.874258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.881633] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=26] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19333935104, fast_delete_log_mode=false) [2025-05-31 02:59:14.885815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.886209] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=87, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:14.886271] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5520, avg_group_log_size=460) [2025-05-31 02:59:14.886294] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5520) [2025-05-31 02:59:14.886375] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:14.889938] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4493-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:14.891980] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5520) [2025-05-31 02:59:14.892074] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=123, avg_log_submit_to_flush_time=4414, avg_log_submit_to_slide_time=5667) [2025-05-31 02:59:14.892522] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5520, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:14.898449] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=33] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:14.900998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.909082] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:14.911282] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678354554142000, v:0}, new_max_version={val:1748678354554142000, v:0}, last_min_version={val:1748678354299974000, v:0}, last_max_version={val:1748678354299974000, v:0}, total_time_us=8306, delta_us=357137) [2025-05-31 02:59:14.911336] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678354554142000, v:0}, min_version={val:1748678354554142000, v:0}, max_version={val:1748678354554142000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:14.911393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.919910] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=28] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:14.921605] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.932001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.938191] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=60] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:14.942986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.949227] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=34] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:14.953600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.962139] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=335] success to update min reserved snapshot(reserved_snapshot=1748676552212516000, duration=1800, snapshot_gc_ts=1748678352212516000) [2025-05-31 02:59:14.964566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.967791] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81740][T1_TimerWK0_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=104] update reserved snapshot finished(cost_ts=5658, reserved_snapshot=1748676552212516000) [2025-05-31 02:59:14.975058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.986069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:14.993529] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA81-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:14.997018] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.003159] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678354766457000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=236692, in_cluster_service=true, cluster_version={val:1748678354554142000, v:0}, min_cluster_version={val:1748678354554142000, v:0}, max_cluster_version={val:1748678354554142000, v:0}, get_cluster_version_err=0, cluster_version_delta=449007, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195391, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=345948391, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:15.007228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.017437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.027707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.037970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.038988] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:15.049072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.050666] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=81] ====== check clog disk timer task ====== [2025-05-31 02:59:15.050716] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=41] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:15.059703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.070264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.078327] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:15.078475] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=49] PNIO [ratelimit] time: 1748678355078471, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008400, add_bytes: 0 [2025-05-31 02:59:15.078528] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=85] current monitor number(seq_=-1) [2025-05-31 02:59:15.078592] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=25] gc stale ls task succ [2025-05-31 02:59:15.083260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.093589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.096580] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:15.097505] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1748678355097500, bytes: 7135728197, bw: 0.145287 MB/s, add_ts: 1007227, add_bytes: 153445 [2025-05-31 02:59:15.106706] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:15.107995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.108406] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [26524][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA2-0-0] [lt=42] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678355140784, keepalive_ts:-1}) [2025-05-31 02:59:15.108461] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [26524][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA2-0-0] [lt=53] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678355140784, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:15.109634] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81744][T1_TimerWK3_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=52] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:15.110073] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 03:59:15.110607] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [28540][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:15.110984] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:15.112168] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3536, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:15.112229] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=796, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:15.112316] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5632) [2025-05-31 02:59:15.112348] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=53, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:15.112678] INFO [LIB] stat (utility.h:1211) [23570][T1_L0_G1][T1][YB42AC10871E-0006366549CC11C1-0-0] [lt=15] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:15.112698] INFO [LIB] stat (utility.h:1211) [23570][T1_L0_G1][T1][YB42AC10871E-0006366549CC11C1-0-0] [lt=19] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=5407, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:15.113703] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:15.122270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.132465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.138870] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=109] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:15.143052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.155131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.162975] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81433][pnio1][T0][YB42AC10870A-000636671AB94762-0-0] [lt=6][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:1301178195, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678355180994, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1387523, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678355156968, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678355162967, enqueue_timestamp_:0, request_arrival_time_:1748678355162967, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:15.163096] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81433][pnio1][T0][YB42AC10870A-000636671AB94762-0-0] [lt=94][errcode=-5150] deliver rpc request fail(&req=0x2b099c804098, ret=-5150) [2025-05-31 02:59:15.171051] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC10870A-000636671538EF54-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1748678355166212000, start_id=1748678355166212000, end_id=1748678355166212001) [2025-05-31 02:59:15.172546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.183503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.196452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.207415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.218089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.230868] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.244303] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=239][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.255723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.270590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.281499] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:15.281796] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4388, iops:14, norm_iops:3, bw:60926, [delay/us]:prepare:13, schedule:151, submit:94, rt:3024, total:3284) [2025-05-31 02:59:15.281987] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=135] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:15.282144] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=133] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4389, avg_iops:14, avg_bw:60900, [delay/us]: prepare:13, schedule:151, submit:94, device:3024, total:3284) [2025-05-31 02:59:15.282201] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:15.282248] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4059, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:400, total:413) [2025-05-31 02:59:15.282287] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:15.282342] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4059, [delay/us]: prepare:0, schedule:0, submit:600000000, device:400, total:413) [2025-05-31 02:59:15.282471] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=109] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:15.282503] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:15.282519] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:15.282534] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:15.282547] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:15.282562] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=141kB/s, net_out=104kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:15.282595] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] successfully alloc block(macro_id={[ver=1,mode=0,seq=399450][2nd=2888]}) [2025-05-31 02:59:15.290209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.300645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:15.302029] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [28578][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:15.302830] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137345722}], scn_array=[cnt:1, {val:1748678355166212001, v:0}], curr_size=122, accum_size=1959, time_cost=10597) [2025-05-31 02:59:15.306642] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:15.310259] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:15.327562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.338025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.339462] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=104] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:15.341186] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2025-05-31 02:59:15.352597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.364078] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.364133] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=66] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.364289] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.364338] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.368342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.378607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.389725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.404249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.406984] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38AB-0-0] [lt=25][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525404589148, cache_obj->added_lc()=false, cache_obj->get_object_id()=53681, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:15.415167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.427432] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83090][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA83-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525404609602, cache_obj->added_lc()=false, cache_obj->get_object_id()=53682, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:15.432006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.432667] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=70] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:15.434602] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=46] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:15.434710] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=22] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:15.444437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.454685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.467509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.477743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.488286] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.489151] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=27] ts response task statistics(alloc_count=486674, free_count=243336) [2025-05-31 02:59:15.489202] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=681, get_gts_with_stc_cnt=36, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=35, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:15.489223] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:15.489280] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678354988218], gts:1748678355023864000, latest_srr:[mts=1748678355489238]}) [2025-05-31 02:59:15.499376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.507754] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678355166212000, v:0}, base_version={val:1748678354903613000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678355303138000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678355507731, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678355166212001, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678355443643, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678355166212000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678355313205, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678350821779000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678355503765, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678350666212000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678355495776, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678350768592000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678355436351, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:59:15.510944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.511154] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:15.511245] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB05-0-0] [lt=150] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.629962968826293945e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=281, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.767437218177893101e+03) [2025-05-31 02:59:15.512761] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=34] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:64K/14:182153 disk:64K/14:182153 t1m2:64K/14:182153 t1m2g0:64K/14:182153 |q0: 0 |q1: 0 [2025-05-31 02:59:15.521374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.533597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.539592] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=61] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:15.541626] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:15.544243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.551122] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38AD-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525404733290, cache_obj->added_lc()=false, cache_obj->get_object_id()=53683, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:15.551727] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=50] ====== checkpoint timer task ====== [2025-05-31 02:59:15.552470] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:15.552517] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=46] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:15.552812] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:15.553014] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=195] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:15.553079] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=58] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:15.553096] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678353762182001, v:0}) [2025-05-31 02:59:15.553105] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:15.553112] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:15.553126] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:15.553147] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:15.553160] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:15.553196] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:15.553219] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:15.553230] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:15.554530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.564799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.572478] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:15.575103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.578609] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:15.585290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.587416] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A72-0-0] [lt=64] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:15.590448] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.590500] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.590559] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.593302] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.593362] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.593405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.593539] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:15.595561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.605827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.618625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=157][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.627313] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D513E-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525404809481, cache_obj->added_lc()=false, cache_obj->get_object_id()=53684, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:15.628979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.641202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.651700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.664399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.674820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.686843] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.699468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=165][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.709976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.711357] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:15.725950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.736236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.742480] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:15.747083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.755552] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:15.755612] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51] temporary file module use no blocks [2025-05-31 02:59:15.755625] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:15.755641] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:15.755657] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525403934426, flush_io_finished_ret_:0, flush_io_finished_round_:19923, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:15.757595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.767883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.778320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.789301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.799973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.810360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.820614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.830821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.841644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=176][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.851899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.863615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.873834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.884468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=158][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.892643] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4494-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:15.894774] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.901146] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=75, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:15.901186] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4092, avg_group_log_size=372) [2025-05-31 02:59:15.901207] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4092) [2025-05-31 02:59:15.901222] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:15.903329] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=230] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4092) [2025-05-31 02:59:15.906109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.906406] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [27077][T1_L0_G1][T1][YB42AC10871E-0006366549CC11C9-0-0] [lt=62] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=107, avg_log_submit_to_flush_time=5392, avg_log_submit_to_slide_time=6566) [2025-05-31 02:59:15.906706] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4092, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:15.912248] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 03:59:15.920267] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [28578][T1_L0_G1][T1][YB42AC1087A7-00063665517F5604-0-0] [lt=69] init point thread id with(&point=0x55c090905c40, idx_=4002, point=[thread id=28578, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=28578) [2025-05-31 02:59:15.920612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.921447] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678355506530000, v:0}, new_max_version={val:1748678355506530000, v:0}, last_min_version={val:1748678355166212001, v:0}, last_max_version={val:1748678355166212001, v:0}, total_time_us=11993, delta_us=414913) [2025-05-31 02:59:15.921487] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678355506530000, v:0}, min_version={val:1748678355506530000, v:0}, max_version={val:1748678355506530000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:15.933322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.943557] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=233] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:15.944287] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.949547] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=29] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:15.954554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.957001] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=60] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:59:15.964801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.977005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.987462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:15.990789] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=59] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:15.990822] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678355489545], gts:1748678355524765000, latest_srr:[mts=1748678355990743]}) [2025-05-31 02:59:15.997720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.001171] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:16.001202] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] try gc retain ctx [2025-05-31 02:59:16.001211] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-31 02:59:16.001291] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678355909177000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678345706851001, v:0}, keep_alive_scn_:{val:1748678350870242001, v:0}, update_ts_:1748678356000186}) [2025-05-31 02:59:16.002138] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:16.007960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.021072] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA82-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:16.021599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.032013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.041514] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:16.042428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.053141] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.064057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.076153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.080688] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=19] current monitor number(seq_=-1) [2025-05-31 02:59:16.080805] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=14] gc stale ls task succ [2025-05-31 02:59:16.082052] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=62] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:16.086089] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1748678356086084, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-05-31 02:59:16.086768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.097818] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=71] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:16.097970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.105142] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1748678356105138, bytes: 7135896029, bw: 0.158844 MB/s, add_ts: 1007638, add_bytes: 167832 [2025-05-31 02:59:16.109123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.109815] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=34] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:16.112714] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:16.119228] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC11CD-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4215) [2025-05-31 02:59:16.119478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.119809] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4492, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:16.119930] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=102] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=964, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:16.120141] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=98] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=41, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:16.120182] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=7214, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:16.120208] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=83, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:16.130108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.140461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.144234] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:16.150722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.160995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.164788] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AFC-0-0] [lt=32] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:16.171217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.173825] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=218] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:16.173934] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=90] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:16.173951] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1045) [2025-05-31 02:59:16.181416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.191682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.202244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.214322] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2073-0-0] [lt=29] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:16.214365] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2073-0-0] [lt=33] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=45) [2025-05-31 02:59:16.217652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.227928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.238202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.248442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.259042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.269924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.280107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.292515] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:16.292578] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48618, [delay/us]:prepare:15, schedule:665, submit:81, rt:3844, total:4606) [2025-05-31 02:59:16.292597] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:16.292658] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48640, [delay/us]: prepare:15, schedule:665, submit:81, device:3844, total:4606) [2025-05-31 02:59:16.292671] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:16.292697] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4053, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1774, total:1790) [2025-05-31 02:59:16.292722] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:16.292751] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4053, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1774, total:1790) [2025-05-31 02:59:16.292769] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:16.292783] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:16.292791] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:16.292799] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:16.292808] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:16.292817] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=168kB/s, net_out=106kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:16.292837] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399451][2nd=2606]}) [2025-05-31 02:59:16.294109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.307685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.307745] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:16.307809] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [25121][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=10] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:16.314838] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=424656, replace_num=15728) [2025-05-31 02:59:16.314944] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:16.316848] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:16.317948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.330108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.340356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.344817] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=156] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:16.353966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.358024] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=34] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:16.358050] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:16.358104] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4A-0-0] [lt=15] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:16.358123] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4A-0-0] [lt=16] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:16.358993] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.359036] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=111] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.359063] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.359088] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.366674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.377012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.387358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.397634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.408057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.418351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.418994] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38AF-0-0] [lt=87][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525405601163, cache_obj->added_lc()=false, cache_obj->get_object_id()=53685, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:16.429063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.439446] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.444028] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:16.449630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.459840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.469595] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:16.472425] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.473821] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=81] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:16.476047] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=97] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:16.477746] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=39] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:16.477788] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=40] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:16.479585] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=11] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:16.479686] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=50] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:16.479704] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=16] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:16.479730] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:16.480397] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=661] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:16.480443] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=36] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:16.480480] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=34] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:16.480508] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=23] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:16.480555] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=46] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:16.482665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.483383] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:16.483430] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=43] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:16.483443] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:16.483478] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:16.483491] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:16.483500] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:16.483512] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:16.483529] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDE-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:16.491379] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:16.493182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.503896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.505965] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:784206 disk:48K/11:784206 t1m2:48K/11:784206 t1m2g0:48K/11:784206 |q0: 0 |q1: 0 [2025-05-31 02:59:16.514235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.515633] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:16.517311] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB14-0-0] [lt=187] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.629962968826293945e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=281, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.767437218177893101e+03) [2025-05-31 02:59:16.524510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.534957] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.544124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=105] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:16.545392] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:16.545469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.557207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.567456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.582314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.582999] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:16.591810] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.591482] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.591958] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.591984] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.591999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.592027] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.592026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:16.592645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.603130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.605793] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:16.605984] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:16.606062] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:16.606165] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:16.613437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.613999] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14] ls blacklist refresh finish(cost_time=39183) [2025-05-31 02:59:16.623620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.633819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.646119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.656323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.666549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.675411] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81742][T1_TimerWK1_Flush][T1][Y0-0000000000000000-0-0] [lt=36] ====== traversal_flush timer task ====== [2025-05-31 02:59:16.675944] INFO [MDS] flush (mds_table_mgr.cpp:184) [81742][T1_TimerWK1_Flush][T1][YB42AC1087A6-0006366543FA5F6A-0-0] [lt=25] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678356516835000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:16.676264] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81742][T1_TimerWK1_Flush][T1][YB42AC1087A6-0006366543FA5F6A-0-0] [lt=257] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:16.677330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.678735] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A7-00063665469F7BF0-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1748678356677467000, start_id=1748678356677467000, end_id=1748678356677467001) [2025-05-31 02:59:16.686257] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D8-0-0] [lt=42] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:16.688196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.701013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.711404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.712484] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [28540][T1_L0_G1][T1][YB42AC10871E-0006366549CC11D2-0-0] [lt=219] init point thread id with(&point=0x55c0909052c0, idx_=3964, point=[thread id=28540, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=28540) [2025-05-31 02:59:16.716069] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=98] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:16.721790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.733403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.744083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.746514] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:16.752561] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:16.755002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.755060] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81744][T1_TimerWK3_ReqMemEvict][T1][YB42AC1087A6-00063665431A60DA-0-0] [lt=26] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=33) [2025-05-31 02:59:16.755453] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678356754634000, start_id=1748678356754634000, end_id=1748678356754634001) [2025-05-31 02:59:16.765155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.775494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.786185] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.787096] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=21] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:16.787143] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=41] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:16.787160] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=313, cache_node_num=312) [2025-05-31 02:59:16.787171] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:16.790289] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=313, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:16.791148] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=855] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:16.797070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.809220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.819785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.830299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.841942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.853928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.859164] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3EC-0-0] [lt=132][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525406041333, cache_obj->added_lc()=false, cache_obj->get_object_id()=53686, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:16.864543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.875013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.885468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.893814] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:16.893916] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=28] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52180160, tenant_id=1) [2025-05-31 02:59:16.894049] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=23] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:16.896265] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4495-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:16.897059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.907299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.910257] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=80] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=79, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:16.910296] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4216, avg_group_log_size=351) [2025-05-31 02:59:16.910317] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4216) [2025-05-31 02:59:16.910332] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:16.913890] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4216) [2025-05-31 02:59:16.915691] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=31, avg_log_gen_to_submit_time=130, avg_log_submit_to_flush_time=5727, avg_log_submit_to_slide_time=6429) [2025-05-31 02:59:16.916363] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4216, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:16.916507] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:16.917699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.921792] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] get number(ret=0, service_type_=0, range=1, base_id=1748678356918069000, start_id=1748678356918069000, end_id=1748678356918069001) [2025-05-31 02:59:16.928130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.928616] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678356718049000, v:0}, new_max_version={val:1748678356718049000, v:0}, last_min_version={val:1748678356491334000, v:0}, last_max_version={val:1748678356491334000, v:0}, total_time_us=9928, delta_us=210564) [2025-05-31 02:59:16.928693] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678356718049000, v:0}, min_version={val:1748678356718049000, v:0}, max_version={val:1748678356718049000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:16.934754] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=126] get number(ret=0, service_type_=0, range=1, base_id=1748678356934181000, start_id=1748678356934181000, end_id=1748678356934181001) [2025-05-31 02:59:16.940722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.942321] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672F2E4728-0-0] [lt=46] get number(ret=0, service_type_=0, range=1, base_id=1748678356941208000, start_id=1748678356941208000, end_id=1748678356941208001) [2025-05-31 02:59:16.947173] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=60] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:16.951056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.951555] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:16.951710] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=12] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-05-31 02:59:16.961289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.971522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.982068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.993017] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678356992232000, start_id=1748678356992232000, end_id=1748678356992232001) [2025-05-31 02:59:16.993698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:16.995249] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD40-0-0] [lt=68] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678356991725, keepalive_ts:-1}) [2025-05-31 02:59:16.995287] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD40-0-0] [lt=37] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678356991725, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:17.004159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.015300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.026107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.037728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.037975] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81065][TimerWK3_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=38] runTimer to refresh locality_info(ret=0) [2025-05-31 02:59:17.041626] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=156] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:59:17.041723] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=81] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:59:17.041739] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:59:17.041762] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:59:17.041785] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=22] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:59:17.041801] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=7] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:59:17.041816] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:59:17.041825] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:59:17.047027] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:17.048178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.050972] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA83-0-0] [lt=36][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:17.051421] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-05-31 02:59:17.051449] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:17.059708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.070469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.080756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.082148] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=29] current monitor number(seq_=-1) [2025-05-31 02:59:17.082229] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:59:17.083281] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=258] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:17.092745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.093119] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678357093114, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007030, add_bytes: 0 [2025-05-31 02:59:17.093292] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=14] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:17.098181] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=102] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:17.104250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.106821] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1748678357106817, bytes: 7136072906, bw: 0.168400 MB/s, add_ts: 1001679, add_bytes: 176877 [2025-05-31 02:59:17.110026] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=43] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:17.119012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.119860] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:17.125538] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4154, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:17.125578] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=502, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:17.125715] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5940) [2025-05-31 02:59:17.125762] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=38, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:17.127395] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC11DA-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=11, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:17.127418] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC11DA-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=6286, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 03:59:17.145670] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [29190][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:17.147763] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40218624, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:17.147954] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:17.160347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.170627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.171609] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=29][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.171657] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=43][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.171677] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f1, hex: 02844EE869360600, int: 1748678357189634}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f9, hex: 02844EE869360600, int: 1748678357189634}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.171949] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:45) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=105] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=60680, insert_dup_release_count=60680, col_map_alloc_count=0, col_map_release_count=0) [2025-05-31 02:59:17.172014] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=40][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.177146] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=38] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:20864}, to_seq_no={branch:0, seq:10885}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332585} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.177266] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:148) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A7-0-0] [lt=62] transaction rpc statistics(total_rt=1895, total_process=10, avg_rt=172) [2025-05-31 02:59:17.181108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.191345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.202660] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A8-0-0] [lt=66] get number(ret=0, service_type_=0, range=1, base_id=1748678357199757000, start_id=1748678357199757000, end_id=1748678357199757001) [2025-05-31 02:59:17.203085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.216030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.226274] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.236673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.244187] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A9-0-0] [lt=14][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.244240] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A9-0-0] [lt=49][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.244312] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A9-0-0] [lt=62][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098e00c820, local_datums:0x2b098e00c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098e00c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b098e00c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4f3, hex: 58894FE869360600, int: 1748678357256536}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4fb, hex: 58894FE869360600, int: 1748678357256536}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.244577] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A9-0-0] [lt=127][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.247335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.257742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.258443] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83074][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4A9-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:36583}, to_seq_no={branch:0, seq:19484}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332590} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.270230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.281380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.292449] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AB-0-0] [lt=12][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.292488] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AB-0-0] [lt=35][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.292506] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AB-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f0, hex: F78350E869360600, int: 1748678357320695}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f8, hex: F78350E869360600, int: 1748678357320695}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.292649] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AB-0-0] [lt=66][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.294290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.295311] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83090][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AB-0-0] [lt=36] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9493}, to_seq_no={branch:0, seq:4341}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332591} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.303412] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:17.303644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=100] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:16, norm_iops:4, bw:72927, [delay/us]:prepare:24, schedule:762, submit:235, rt:2777, total:3799) [2025-05-31 02:59:17.303668] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:17.303726] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:16, avg_bw:72920, [delay/us]: prepare:24, schedule:762, submit:235, device:2777, total:3799) [2025-05-31 02:59:17.303740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:17.303767] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4051, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:961, total:976) [2025-05-31 02:59:17.303792] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:17.303820] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4051, [delay/us]: prepare:0, schedule:0, submit:600000000, device:961, total:976) [2025-05-31 02:59:17.303838] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:17.303851] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:17.303860] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:17.303868] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:17.303956] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:17.303967] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=287kB/s, net_out=182kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:17.303987] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399452][2nd=2337]}) [2025-05-31 02:59:17.304762] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AC-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1748678357303347000, start_id=1748678357303347000, end_id=1748678357303347001) [2025-05-31 02:59:17.306409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.318635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.324426] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:17.328944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.330008] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:17.335958] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AD-0-0] [lt=32][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.336007] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AD-0-0] [lt=44][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.336027] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AD-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09a9e0c820, local_datums:0x2b09a9e0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09a9e0c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09a9e0c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c4f0, hex: 81E550E869360600, int: 1748678357345665}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c4f8, hex: 81E550E869360600, int: 1748678357345665}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09a9e0c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.336199] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AD-0-0] [lt=99][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.338531] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AD-0-0] [lt=11] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:27547}, to_seq_no={branch:0, seq:21495}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332592} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.339456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.347408] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33D3-0-0] [lt=56][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525406529577, cache_obj->added_lc()=false, cache_obj->get_object_id()=53687, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:17.348467] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-38121472, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:17.350145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.350375] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2025-05-31 02:59:17.360542] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.360590] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.360619] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.360645] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.360893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.367941] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AF-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.367980] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AF-0-0] [lt=33][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.367998] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AF-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f1, hex: A0AE51E869360600, int: 1748678357397152}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f9, hex: A0AE51E869360600, int: 1748678357397152}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.368153] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4AF-0-0] [lt=65][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.370112] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83080][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4AF-0-0] [lt=35] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7145}, to_seq_no={branch:0, seq:3357}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332593} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.371125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.376025] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=52] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:17.381355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:17.388320] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [29190][T1_L0_G1][T1][YB42AC10871E-0006366549CC11E1-0-0] [lt=44] init point thread id with(&point=0x55c0908cf540, idx_=518, point=[thread id=29190, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=29190) [2025-05-31 02:59:17.393057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.405149] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=46][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aeeb40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332594} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3882} write_epoch=1, prev=this=0x2b095e56b6e8 trans_version={val:1748677079237279001, v:0} scn={val:1748677079237279001, v:0} tx_id={txid:11330828} prev=0x2b095e56b530 next=(nil) modify_count=1 acc_checksum=1298508093 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:31079} write_epoch=3, snapshot_version={val:1748678357390523000, v:0}, *this={this=0x2b09b3155250 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e56b6e8 latest_compact_node=(nil) max_trans_version={val:1748678296453168000, v:0} max_trans_id=11332510 max_elr_trans_version={val:1748678296453168000, v:0} max_elr_trans_id=11332510 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748677079237279001, v:0} min_modify_scn={val:1748677079237279001, v:0}}) [2025-05-31 02:59:17.405282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.405256] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=104][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aeeb40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332594} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3882} write_epoch=1, ctx={this:0x2b091ca551a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387301804, tablet_id:{id:309}, table_iter:0x2b091ca536d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387301804, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357390523000, v:0}, tx_id:{txid:11332594}, scn:{branch:0, seq:3879}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332594}, tx_desc:{this:0x2b0992a57900, tx_id:{txid:11332594}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357401642, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357426456, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:3882}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.405356] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=98][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332594}, data:0x2b091ca520c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3882}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.405373] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332594} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.405426] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=50][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.405444] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b093a60c820, local_datums:0x2b093a60c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b093a60c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b093a60c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c4f1, hex: 192152E869360600, int: 1748678357426457}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c4f9, hex: 192152E869360600, int: 1748678357426457}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c501, hex: 8EC8A20200000000, int: 44222606}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b093a60c509, hex: 0080730300000000, int: 57901056}]}) [2025-05-31 02:59:17.405553] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83091][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.417224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.418758] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B1-0-0] [lt=52] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:22252}, to_seq_no={branch:0, seq:3880}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332594} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.427581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.433203] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38B1-0-0] [lt=290][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525406615372, cache_obj->added_lc()=false, cache_obj->get_object_id()=53688, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:17.436715] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83087][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B2-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678357434624000, start_id=1748678357434624000, end_id=1748678357434624001) [2025-05-31 02:59:17.438706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.453246] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b098034ab40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332595} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:8202} write_epoch=1, prev=this=0x2b0980338068 trans_version={val:1748677870245259000, v:0} scn={val:1748677870245259000, v:0} tx_id={txid:11331923} prev=0x2b0988a82038 next=(nil) modify_count=11 acc_checksum=2681614678 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:33914} write_epoch=0, snapshot_version={val:1748678357441526000, v:0}, *this={this=0x2b09b31554f0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980338068 latest_compact_node=0x2b0980338068 max_trans_version={val:1748678296495933000, v:0} max_trans_id=11332511 max_elr_trans_version={val:1748678296495933000, v:0} max_elr_trans_id=11332511 latest_compact_ts=1748677870256920 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677870245259000, v:0} min_modify_scn={val:1748675800888657000, v:0}}) [2025-05-31 02:59:17.453319] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=71][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b098034ab40 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332595} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:8202} write_epoch=1, ctx={this:0x2b091c1cf1a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387346670, tablet_id:{id:309}, table_iter:0x2b091c1cd6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387346670, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357441526000, v:0}, tx_id:{txid:11332595}, scn:{branch:0, seq:8199}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332595}, tx_desc:{this:0x2b0991bdc100, tx_id:{txid:11332595}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357452107, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357475689, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c556d0, mem_ctx:0x2b0904c56c50, tx_scn:{branch:0, seq:8202}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.453500] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=172][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332595}, data:0x2b091c1cc0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:8202}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.453537] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=35][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332595} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.453570] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.453585] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099ee0c820, local_datums:0x2b099ee0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b099ee0c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b099ee0c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4f1, hex: 6AE152E869360600, int: 1748678357475690}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4f9, hex: 6AE152E869360600, int: 1748678357475690}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c501, hex: DC1B770000000000, int: 7805916}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c509, hex: 00F0B50000000000, int: 11923456}]}) [2025-05-31 02:59:17.453701] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.454138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.455910] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83095][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B3-0-0] [lt=40] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14716}, to_seq_no={branch:0, seq:8200}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332595} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.464910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.475334] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.481655] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B5-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.481690] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B5-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.481708] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B5-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f3, hex: 8C7653E869360600, int: 1748678357513868}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4fb, hex: 8C7653E869360600, int: 1748678357513868}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.481968] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B5-0-0] [lt=181][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.482991] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B5-0-0] [lt=34] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4095}, to_seq_no={branch:0, seq:3003}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332596} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.486045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.496250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.496953] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=43] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678357496854], gts:1748678357496907000, latest_srr:[mts=1748678357496854]}) [2025-05-31 02:59:17.496989] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:17.497550] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83094][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B6-0-0] [lt=27] get number(ret=0, service_type_=0, range=1, base_id=1748678357496907000, start_id=1748678357496907001, end_id=1748678357496907002) [2025-05-31 02:59:17.506979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.508735] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B7-0-0] [lt=14][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.508766] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B7-0-0] [lt=26][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.508784] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B7-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f3, hex: 47D953E869360600, int: 1748678357539143}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4fb, hex: 47D953E869360600, int: 1748678357539143}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.509021] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4B7-0-0] [lt=144][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.510066] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4B7-0-0] [lt=106] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6157}, to_seq_no={branch:0, seq:3699}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332597} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.517323] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.527225] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:17.528256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.530078] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=86] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:100K/22:832561 disk:100K/22:832561 t1m2:100K/22:832561 t1m2g0:100K/22:832561 |q0: 0 |q1: 0 [2025-05-31 02:59:17.539292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.546284] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=36][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aef658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332599} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5999} write_epoch=1, prev=this=0x2b0980347c18 trans_version={val:1748678296635244000, v:0} scn={val:1748678296635244000, v:0} tx_id={txid:11332515} prev=0x2b0988ae1878 next=(nil) modify_count=115 acc_checksum=2759535894 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:16776} write_epoch=3, snapshot_version={val:1748678357540334000, v:0}, *this={this=0x2b09b3155ca8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0980347c18 latest_compact_node=0x2b09803451d8 max_trans_version={val:1748678296635244000, v:0} max_trans_id=11332515 max_elr_trans_version={val:1748678296635244000, v:0} max_elr_trans_id=11332515 latest_compact_ts=1748678235902489 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678296635244000, v:0} min_modify_scn={val:1748674824325484000, v:0}}) [2025-05-31 02:59:17.546390] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=103][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aef658 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332599} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5999} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387441427, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387441427, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357540334000, v:0}, tx_id:{txid:11332599}, scn:{branch:0, seq:5996}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332599}, tx_desc:{this:0x2b094e8455a0, tx_id:{txid:11332599}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357545103, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357575514, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:5999}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.546578] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=186][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332599}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:5999}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.546598] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332599} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.546630] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.546645] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f1, hex: 5B6754E869360600, int: 1748678357575515}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f9, hex: 5B6754E869360600, int: 1748678357575515}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c501, hex: 2991760D00000000, int: 225874217}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c509, hex: 2991760D00000000, int: 225874217}]}) [2025-05-31 02:59:17.546757] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.547517] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:17.547958] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=28] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7381}, to_seq_no={branch:0, seq:5997}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332599} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.549618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.551082] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:17.555574] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83104][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BB-0-0] [lt=30] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.581963014602661133e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=279, Throttle Trigger(MB)=167, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.863002783878612718e+03) [2025-05-31 02:59:17.561029] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83097][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BC-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1748678357560416000, start_id=1748678357560416000, end_id=1748678357560416001) [2025-05-31 02:59:17.561491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.573095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.577951] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aefab0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332600} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7295} write_epoch=1, prev=this=0x2b0988ae1cb8 trans_version={val:1748678296679951000, v:0} scn={val:1748678296679951000, v:0} tx_id={txid:11332516} prev=0x2b0988ae1b00 next=(nil) modify_count=115 acc_checksum=2772169951 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:15369} write_epoch=3, snapshot_version={val:1748678357572231000, v:0}, *this={this=0x2b09b3156068 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae1cb8 latest_compact_node=0x2b0980345360 max_trans_version={val:1748678296679951000, v:0} max_trans_id=11332516 max_elr_trans_version={val:1748678296679951000, v:0} max_elr_trans_id=11332516 latest_compact_ts=1748678235966369 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678296679951000, v:0} min_modify_scn={val:1748674824382632000, v:0}}) [2025-05-31 02:59:17.578016] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=63][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aefab0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332600} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:7295} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387471800, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387471800, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357572231000, v:0}, tx_id:{txid:11332600}, scn:{branch:0, seq:7292}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332600}, tx_desc:{this:0x2b094e86bcc0, tx_id:{txid:11332600}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357576995, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357603731, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:7295}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.578143] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=126][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332600}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:7295}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.578162] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332600} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.578193] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.578208] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f1, hex: 45DA54E869360600, int: 1748678357604933}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f9, hex: 45DA54E869360600, int: 1748678357604933}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c501, hex: 30B8BA0900000000, int: 163231792}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c509, hex: 30B8BA0900000000, int: 163231792}]}) [2025-05-31 02:59:17.578465] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.584500] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BD-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:12833}, to_seq_no={branch:0, seq:7293}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332600} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.585081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.585290] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:17.590948] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.591085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.591122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.591498] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=367] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.591557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.591748] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.592043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:17.594281] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83099][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BE-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1748678357594144000, start_id=1748678357594144000, end_id=1748678357594144001) [2025-05-31 02:59:17.596049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.598396] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81064][TimerWK2_ServerTracerTim][T0][YB42AC1087A6-0006366542DA5482-0-0] [lt=28] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:17.604340] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BF-0-0] [lt=10][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.604421] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BF-0-0] [lt=75][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.604444] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BF-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099ee0c820, local_datums:0x2b099ee0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b099ee0c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b099ee0c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4f2, hex: 8C4E55E869360600, int: 1748678357634700}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c4fa, hex: 8C4E55E869360600, int: 1748678357634700}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099ee0c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.604612] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83073][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BF-0-0] [lt=69][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.606411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.606584] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4BF-0-0] [lt=45] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7457}, to_seq_no={branch:0, seq:1991}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332601} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.616612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.626404] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83085][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C1-0-0] [lt=113][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.626443] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83085][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C1-0-0] [lt=35][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.626461] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83085][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C1-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099360c820, local_datums:0x2b099360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b099360c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b099360c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c4f1, hex: 0BA155E869360600, int: 1748678357655819}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c4f9, hex: 0BA155E869360600, int: 1748678357655819}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099360c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.626692] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83085][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C1-0-0] [lt=80][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.627988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.629036] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C1-0-0] [lt=69] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8011}, to_seq_no={branch:0, seq:2291}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332602} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.638769] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.652570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.654715] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988aeffb8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332603} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8019} write_epoch=1, prev=this=0x2b095e56d228 trans_version={val:1748677079545868000, v:0} scn={val:1748677079545868000, v:0} tx_id={txid:11330837} prev=0x2b095e56d070 next=(nil) modify_count=1 acc_checksum=4119733741 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:30711} write_epoch=3, snapshot_version={val:1748678357643328000, v:0}, *this={this=0x2b09b3156a10 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e56d228 latest_compact_node=(nil) max_trans_version={val:1748678296856650000, v:0} max_trans_id=11332519 max_elr_trans_version={val:1748678296856650000, v:0} max_elr_trans_id=11332519 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748677079545868000, v:0} min_modify_scn={val:1748677079545868000, v:0}}) [2025-05-31 02:59:17.654779] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988aeffb8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332603} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:8019} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387547553, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387547553, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357643328000, v:0}, tx_id:{txid:11332603}, scn:{branch:0, seq:5363}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332603}, tx_desc:{this:0x2b094e97efc0, tx_id:{txid:11332603}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357653160, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357678565, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:8019}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.654943] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=161][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332603}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:8019}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.654964] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332603} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.654995] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.655011] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f2, hex: E6F955E869360600, int: 1748678357678566}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4fa, hex: E6F955E869360600, int: 1748678357678566}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c502, hex: 0040300400000000, int: 70270976}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c50a, hex: 0040300400000000, int: 70270976}]}) [2025-05-31 02:59:17.655173] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.661184] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C3-0-0] [lt=77] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14827}, to_seq_no={branch:0, seq:5364}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332603} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.662753] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC10870A-0006366715B99E1D-0-0] [lt=31] get number(ret=0, service_type_=0, range=1, base_id=1748678357662521000, start_id=1748678357662521000, end_id=1748678357662521001) [2025-05-31 02:59:17.665314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.676505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.686903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.697158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.700791] INFO [STORAGE] get_kept_snapshot_info (ob_tablet.cpp:5979) [81743][T1_TimerWK2_SSTableGC][T1][YB42AC1087A6-00063665430A5EE7-0-0] [lt=22] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, snapshot_info={type:"SNAPSHOT_ON_TABLET", snapshot:1748671375638251000}, old_snapshot_info={type:"UNDO_RETENTION", snapshot:1748676552212516000}, min_medium_snapshot=9223372036854775807, min_reserved_snapshot_on_ls=1748676552212516000) [2025-05-31 02:59:17.705815] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=26][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af0238 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332604} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:3926} write_epoch=1, prev=this=0x2b0988a83388 trans_version={val:1748677870772395000, v:0} scn={val:1748677870772395000, v:0} tx_id={txid:11331932} prev=0x2b0988a831d0 next=(nil) modify_count=9 acc_checksum=919988878 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:58878} write_epoch=3, snapshot_version={val:1748678357703233000, v:0}, *this={this=0x2b09b3156cb0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b0988a83388 latest_compact_node=0x2b0980307dd8 max_trans_version={val:1748678296890441000, v:0} max_trans_id=11332520 max_elr_trans_version={val:1748678296890441000, v:0} max_elr_trans_id=11332520 latest_compact_ts=1748676470879959 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677870772395000, v:0} min_modify_scn={val:1748675801154435000, v:0}}) [2025-05-31 02:59:17.705936] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=118][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af0238 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332604} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:3926} write_epoch=1, ctx={this:0x2b091cd511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387602427, tablet_id:{id:309}, table_iter:0x2b091cd4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387602427, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357703233000, v:0}, tx_id:{txid:11332604}, scn:{branch:0, seq:3923}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332604}, tx_desc:{this:0x2b0982b5e9c0, tx_id:{txid:11332604}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357704291, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357736021, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:3926}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.706046] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=107][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332604}, data:0x2b091cd4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3926}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.706121] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=72][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332604} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.706192] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.706210] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098420c820, local_datums:0x2b098420c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098420c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b098420c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4f2, hex: 56DA56E869360600, int: 1748678357736022}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4fa, hex: 56DA56E869360600, int: 1748678357736022}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c502, hex: 0090400100000000, int: 21008384}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c50a, hex: 0090400100000000, int: 21008384}]}) [2025-05-31 02:59:17.706419] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=111][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.707831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.710950] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83105][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C5-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7972}, to_seq_no={branch:0, seq:3924}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332604} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.719729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.730684] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:17.731525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.735171] WDIAG [MDS] ~ObOccamTimeGuard (ob_occam_time_guard.h:270) [81743][T1_TimerWK2_SSTableGC][T1][YB42AC1087A6-00063665430A5EE7-0-0] [lt=38][errcode=0] cost too much time:mds_dump_node.h:int oceanbase::storage::mds::MdsDumpNode::init(const uint8_t, const uint8_t, const UserMdsNode &, oceanbase::common::ObIAllocator &) [K = oceanbase::compaction::ObMediumCompactionInfoKey, V = oceanbase::compac..., (this=|threshold=1.00ms|start at 03:59:17.734|219=1us|238=1.15ms|total=1.19ms) [2025-05-31 02:59:17.745527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.753126] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-33906688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:17.758084] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:17.758125] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-05-31 02:59:17.758136] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:17.758205] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=66] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:17.758222] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525405938258, flush_io_finished_ret_:0, flush_io_finished_round_:19925, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:17.760942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.772860] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C9-0-0] [lt=15][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.772947] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C9-0-0] [lt=83][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.772966] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C9-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c780c820, local_datums:0x2b09c780c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c780c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c780c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f1, hex: 76BA57E869360600, int: 1748678357793398}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c4f9, hex: 76BA57E869360600, int: 1748678357793398}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c780c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.773166] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4C9-0-0] [lt=108][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.779417] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.783922] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4C9-0-0] [lt=47] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:22440}, to_seq_no={branch:0, seq:3222}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332606} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.791132] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.806050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.818631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.827771] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=352] get_max_applied_scn(scn={val:1748678357726397000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137369646}, palf_committed_end_scn_:{val:1748678357816205000, v:0}, last_check_scn_:{val:1748678357816205000, v:0}, max_applied_cb_scn_:{val:1748678357726397000, v:0}}) [2025-05-31 02:59:17.827934] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=154] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137369646}, readable_log_tail:{lsn:2137369646}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9982930, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:17.830750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=340][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.837976] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af0ad0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332607} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5876} write_epoch=1, prev=this=0x2b095e56dd48 trans_version={val:1748677079660657000, v:0} scn={val:1748677079660657000, v:0} tx_id={txid:11330841} prev=0x2b095e56db90 next=(nil) modify_count=3 acc_checksum=400322214 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:23697} write_epoch=3, snapshot_version={val:1748678357829358001, v:0}, *this={this=0x2b09802c6900 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b095e56dd48 latest_compact_node=(nil) max_trans_version={val:1748678296968613000, v:0} max_trans_id=11332523 max_elr_trans_version={val:1748678296968613000, v:0} max_elr_trans_id=11332523 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748677079660657000, v:0} min_modify_scn={val:1748674886026790000, v:0}}) [2025-05-31 02:59:17.838135] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=156][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af0ad0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332607} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5876} write_epoch=1, ctx={this:0x2b091cd511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387733474, tablet_id:{id:309}, table_iter:0x2b091cd4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387733474, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357829358001, v:0}, tx_id:{txid:11332607}, scn:{branch:0, seq:5873}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332607}, tx_desc:{this:0x2b0982bb3980, tx_id:{txid:11332607}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357836263, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357862478, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:5876}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.838256] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=119][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332607}, data:0x2b091cd4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:5876}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.838276] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332607} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.838370] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=91][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.838386] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098420c820, local_datums:0x2b098420c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098420c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b098420c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4f1, hex: 4FC858E869360600, int: 1748678357862479}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c4f9, hex: 4FC858E869360600, int: 1748678357862479}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c501, hex: 00600B0000000000, int: 745472}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098420c509, hex: 00600B0000000000, int: 745472}]}) [2025-05-31 02:59:17.838488] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83100][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=58][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.840781] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CB-0-0] [lt=46] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13886}, to_seq_no={branch:0, seq:5874}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332607} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.840890] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.841119] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:113) [81743][T1_TimerWK2_SSTableGC][T1][YB42AC1087A6-00063665430A5EE7-0-0] [lt=107] SSTableGCTask(cost_ts=140750) [2025-05-31 02:59:17.851369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.862003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.865982] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=14][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af0d50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332608} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3302} write_epoch=1, prev=this=0x2b0988ae2ff8 trans_version={val:1748678296994815000, v:0} scn={val:1748678296994815000, v:0} tx_id={txid:11332524} prev=0x2b0988ae2e40 next=(nil) modify_count=115 acc_checksum=2164681596 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:18091} write_epoch=3, snapshot_version={val:1748678357861144000, v:0}, *this={this=0x2b09b3157460 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae2ff8 latest_compact_node=0x2b09803454f8 max_trans_version={val:1748678296994815000, v:0} max_trans_id=11332524 max_elr_trans_version={val:1748678296994815000, v:0} max_elr_trans_id=11332524 latest_compact_ts=1748678236176306 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678296994815000, v:0} min_modify_scn={val:1748674825031954000, v:0}}) [2025-05-31 02:59:17.866273] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=288][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af0d50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332608} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3302} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387764067, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387764067, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357861144000, v:0}, tx_id:{txid:11332608}, scn:{branch:0, seq:3299}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332608}, tx_desc:{this:0x2b094e9ad860, tx_id:{txid:11332608}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357865569, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357897908, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c73650, mem_ctx:0x2b0904c74bd0, tx_scn:{branch:0, seq:3302}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.866454] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=178][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332608}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3302}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.866476] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332608} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.866509] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.866524] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b092360c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f1, hex: B55259E869360600, int: 1748678357897909}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f9, hex: B55259E869360600, int: 1748678357897909}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c501, hex: 00803A0000000000, int: 3833856}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c509, hex: 00803A0000000000, int: 3833856}]}) [2025-05-31 02:59:17.866637] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.870141] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CD-0-0] [lt=117] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4914}, to_seq_no={branch:0, seq:3300}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332608} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.872276] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.880807] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83082][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4CE-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1748678357878957000, start_id=1748678357878957000, end_id=1748678357878957001) [2025-05-31 02:59:17.882663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.893471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.899474] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4496-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:17.905834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.909182] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D1-0-0] [lt=41][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.909216] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D1-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.909233] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D1-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b098e00c820, local_datums:0x2b098e00c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b098e00c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b098e00c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4f1, hex: CBEB59E869360600, int: 1748678357937099}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c4f9, hex: CBEB59E869360600, int: 1748678357937099}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c501, hex: 00C0BC0000000000, int: 12369920}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b098e00c509, hex: 00C0BC0000000000, int: 12369920}]}) [2025-05-31 02:59:17.909369] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D1-0-0] [lt=66][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.911055] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D1-0-0] [lt=61] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:9523}, to_seq_no={branch:0, seq:4984}, remove_cnt=1, elapsed=1127, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332610} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.916391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.921338] INFO [LIB] stat (utility.h:1211) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D2-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=70, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:17.921518] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D2-0-0] [lt=79] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=31, avg_log_batch_cnt=1, total_group_log_size=20022, avg_group_log_size=645) [2025-05-31 02:59:17.921544] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D2-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=19583) [2025-05-31 02:59:17.921560] INFO [LIB] stat (utility.h:1211) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D2-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:17.922822] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=20022) [2025-05-31 02:59:17.925684] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665517F562E-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=31, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=105, avg_log_submit_to_flush_time=3537, avg_log_submit_to_slide_time=4674) [2025-05-31 02:59:17.926949] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=20022, hit_cnt=31, read_cnt=31, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:17.927029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.931055] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:17.933059] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af1420 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332611} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:2184} write_epoch=1, prev=this=0x2b0980338988 trans_version={val:1748677871066493000, v:0} scn={val:1748677871066493000, v:0} tx_id={txid:11331939} prev=0x2b0988a84308 next=(nil) modify_count=7 acc_checksum=2107137994 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678357119642000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:48124} write_epoch=3, snapshot_version={val:1748678357931248000, v:0}, *this={this=0x2b09b3157d40 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0980338988 latest_compact_node=0x2b0980319da8 max_trans_version={val:1748678297067598000, v:0} max_trans_id=11332527 max_elr_trans_version={val:1748678297067598000, v:0} max_elr_trans_id=11332527 latest_compact_ts=1748677079768432 last_compact_cnt=5 total_trans_node_cnt=8 max_modify_scn={val:1748677871066493000, v:0} min_modify_scn={val:1748675801327603000, v:0}}) [2025-05-31 02:59:17.933214] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=150][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af1420 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11332611} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:2184} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678387831191, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678387831191, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678357931248000, v:0}, tx_id:{txid:11332611}, scn:{branch:0, seq:2181}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11332611}, tx_desc:{this:0x2b094e8a26e0, tx_id:{txid:11332611}, state:1, addr:"172.16.135.10:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678357931248, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678357965109, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:2184}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:17.933385] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=169][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11332611}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:2184}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:17.933406] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=20][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.10", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":10},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11332611} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:17.933438] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.933451] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b092360c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f1, hex: 36595AE869360600, int: 1748678357965110}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f9, hex: 36595AE869360600, int: 1748678357965110}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c501, hex: 00A08A0000000000, int: 9084928}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c509, hex: 00A08A0000000000, int: 9084928}]}) [2025-05-31 02:59:17.933549] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.934501] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=88] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678357677829000, v:0}, new_max_version={val:1748678357677829000, v:0}, last_min_version={val:1748678357517969000, v:0}, last_max_version={val:1748678357517969000, v:0}, total_time_us=6612, delta_us=256669) [2025-05-31 02:59:17.934607] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678357677829000, v:0}, min_version={val:1748678357677829000, v:0}, max_version={val:1748678357677829000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:17.937654] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83092][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D3-0-0] [lt=21] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5720}, to_seq_no={branch:0, seq:2182}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332611} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.938130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.948518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.951785] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=42] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:17.954165] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:17.955471] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=18] rpc statistics(total_trans_msg_count=43, total_batch_msg_count=0) [2025-05-31 02:59:17.960063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.960286] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D5-0-0] [lt=12][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:17.960342] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D5-0-0] [lt=50][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:17.960363] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D5-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E3130, cstr: 172.16.135.10}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d9, hex: 0A00000000000000, int: 10}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b092360c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f0, hex: 8AC25AE869360600, int: 1748678357992074}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f8, hex: 8AC25AE869360600, int: 1748678357992074}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:17.960605] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC10870A-0006366715F8F4D5-0-0] [lt=141][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:17.964448] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83103][T1_L0_G0][T1][YB42AC10870A-0006366715F8F4D5-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5504}, to_seq_no={branch:0, seq:2969}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11332612} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:17.970290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.977955] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28] LogLoopThread round_cost_time(us)(round_cost_time=19) [2025-05-31 02:59:17.980595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:17.990923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.004288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.009634] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=53] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:18.014622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.025066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.035357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.049140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.049954] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:18.056541] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=35] [APPLY STAT CB APPEND COST TIME](cur_stat_count=72, stat_interval=5000000, avg (count/cost)=152, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:18.056578] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=36] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=72, stat_interval=5000000, avg (count/cost)=6455, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:18.056587] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=72, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:18.056594] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=72, stat_interval=5000000, avg (count/cost)=-988, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:18.056601] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38AA-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=72, stat_interval=5000000, avg (count/cost)=5619, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:18.059991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.070250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.076752] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA84-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:18.081931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.082423] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:59:18.082501] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=69] gc stale ls task succ [2025-05-31 02:59:18.086979] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:18.092329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.099808] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678358099802, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006688, add_bytes: 0 [2025-05-31 02:59:18.101213] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=34] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:18.102674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.110165] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:18.112777] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1748678358112773, bytes: 7136906101, bw: 0.789892 MB/s, add_ts: 1005956, add_bytes: 833195 [2025-05-31 02:59:18.112948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.123324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.123360] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:18.124709] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [23570][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=7] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:18.131704] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=169] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:18.133503] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC11F9-0-0] [lt=40] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=20030) [2025-05-31 02:59:18.133663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.134321] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=70] [PALF STAT WRITE LOG COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=2616, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:18.134372] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=646, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:18.135541] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=16, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:18.135581] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=4620, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:18.135606] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FLUSH CB COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=41, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:18.144994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.155385] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:18.155570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.166346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.176977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.187179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.199740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.210293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.222587] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2074-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:18.222628] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2074-0-0] [lt=33] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=44) [2025-05-31 02:59:18.222828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.233549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.244031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.254716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.265099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.275822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.287012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.294351] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA5484-0-0] [lt=85] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678353655442, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:18.294454] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA5484-0-0] [lt=50] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678353657981, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:18.294647] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81064][TimerWK2_DumpLSLoc][T0][YB42AC1087A6-0006366542DA5484-0-0] [lt=64] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678353666305, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678353666541, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:18.300598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.312027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.322536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.322562] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:18.322636] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4827, iops:27, norm_iops:8, bw:132636, [delay/us]:prepare:12, schedule:549, submit:146, rt:1800, total:2508) [2025-05-31 02:59:18.322655] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:18.322712] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4827, avg_iops:28, avg_bw:132649, [delay/us]: prepare:12, schedule:549, submit:146, device:1800, total:2508) [2025-05-31 02:59:18.322725] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:18.322750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4019, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:591, total:661) [2025-05-31 02:59:18.322774] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:18.322802] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4019, [delay/us]: prepare:0, schedule:0, submit:600000000, device:591, total:661) [2025-05-31 02:59:18.322820] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:18.322834] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:18.322842] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:18.322942] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=88] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:18.322958] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:18.322968] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SERVER] net_in=701kB/s, net_out=314kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:18.322990] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] successfully alloc block(macro_id={[ver=1,mode=0,seq=399453][2nd=2619]}) [2025-05-31 02:59:18.328603] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4251-0-0] [lt=63] get number(ret=0, service_type_=0, range=1, base_id=1748678358325771000, start_id=1748678358325771000, end_id=1748678358325771001) [2025-05-31 02:59:18.332011] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:18.333212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.345077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.355394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.356485] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40218624, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:18.358154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.361728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.361770] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.361799] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.362489] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=25] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:18.362567] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=77] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:18.362698] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4B-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:18.362726] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4B-0-0] [lt=24] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:18.365626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.376474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.388722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.400486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.411465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.421936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.433661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.443361] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=41] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:18.443965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.445498] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=56] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:18.445625] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=43] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:18.451395] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83084][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38B3-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525407633565, cache_obj->added_lc()=false, cache_obj->get_object_id()=53689, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:18.451902] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:18.455320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.465441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.476735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.487068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.498968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.500734] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13] ts response task statistics(alloc_count=486746, free_count=243372) [2025-05-31 02:59:18.500825] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=80] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:18.501016] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=95] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678358000016], gts:1748678358038131000, latest_srr:[mts=1748678358500924]}) [2025-05-31 02:59:18.509927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.521167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.531502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.533457] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=440384, replace_num=15728) [2025-05-31 02:59:18.533490] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 03:59:18.534154] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [29559][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:18.534206] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:18.538015] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:18.547007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.550340] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:18.558826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.559356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:18.569661] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83084][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38B5-0-0] [lt=54][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525407751829, cache_obj->added_lc()=false, cache_obj->get_object_id()=53690, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:18.569940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.571252] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38B7-0-0] [lt=46][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:18.576361] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38B9-0-0] [lt=26][errcode=-4201] hashtable not init [2025-05-31 02:59:18.576646] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38BA-0-0] [lt=28] get number(ret=0, service_type_=0, range=1, base_id=1748678358576593000, start_id=1748678358576593000, end_id=1748678358576593001) [2025-05-31 02:59:18.576985] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=35] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:108K/24:276378 disk:108K/24:276378 t1m2:108K/24:276378 t1m2g0:108K/24:276378 |q0: 0 |q1: 0 [2025-05-31 02:59:18.582197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.587386] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:18.587498] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83085][T1_L0_G0][T1][YB42AC108714-000636671E2D3A73-0-0] [lt=15] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:18.591203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.592599] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.592645] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.592674] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.594156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.594284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.594321] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.594389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:18.605261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:18.606046] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [29577][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:18.607574] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:59:18.617882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.628110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.638363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.648754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.659083] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.669660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.681996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.683039] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=28] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:18.683103] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81740][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=48] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=966) [2025-05-31 02:59:18.693766] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20D9-0-0] [lt=21] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:18.693989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.704182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.714420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.727520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.732667] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB31-0-0] [lt=115] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.557963609695434570e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=277, Throttle Trigger(MB)=166, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.912331360244965708e+03) [2025-05-31 02:59:18.736066] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC10871E-000636654E089A44-0-0] [lt=32][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525407918237, cache_obj->added_lc()=false, cache_obj->get_object_id()=53691, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:18.736799] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:18.738591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.739355] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137377348}], scn_array=[cnt:1, {val:1748678358576593001, v:0}], curr_size=122, accum_size=5255, time_cost=12337) [2025-05-31 03:59:18.742745] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [29559][T1_L0_G1][T1][YB42AC10871E-0006366549CC1200-0-0] [lt=194] init point thread id with(&point=0x55c0908d5180, idx_=887, point=[thread id=29559, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=29559) [2025-05-31 02:59:18.750626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.758553] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:18.758595] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-31 02:59:18.758605] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:18.758622] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:18.758653] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525406940437, flush_io_finished_ret_:0, flush_io_finished_round_:19926, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:18.759102] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=87] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:18.761551] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:18.761733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.772082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.782373] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.792810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.803201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.806493] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:18.806662] WDIAG read_multi_row (ob_table_access_helper.h:238) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:18.808692] INFO delete_row (ob_table_access_helper.h:339) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678358806681) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678358806681) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:59:18.809395] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=20] ====== tenant freeze timer task ====== [2025-05-31 02:59:18.809424] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=20] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52180160, tenant_id=1) [2025-05-31 02:59:18.809446] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=12] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:18.814930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.827658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.837891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.850253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.860524] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.873164] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.883491] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.893773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.900976] INFO [SERVER] refresh_avail_memory (ob_table_load_assigned_memory_manager.cpp:115) [81743][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=21] ObTableLoadAssignedMemoryManager::refresh_avail_memory(MTL_ID()=1, avail_memory=53687091, chunk_count_=0, avail_sort_memory_=53687091, avail_memory_=53687091) [2025-05-31 02:59:18.902137] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4497-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:18.905655] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.916248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.922716] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81743][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=38] clear dtl interm result cost(us)(clear_cost=4344, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:59:18.926395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.928338] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=86, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:18.928393] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5966, avg_group_log_size=458) [2025-05-31 02:59:18.928416] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=6405) [2025-05-31 02:59:18.928431] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:18.930229] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5966) [2025-05-31 02:59:18.933151] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [28578][T1_L0_G1][T1][YB42AC1087A7-00063665517F563B-0-0] [lt=63] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=610, avg_log_gen_to_submit_time=718, avg_log_submit_to_flush_time=5866, avg_log_submit_to_slide_time=6125) [2025-05-31 02:59:18.935505] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=50] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:59:18.939013] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5966, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:18.939217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.941652] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:18.944448] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=112] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678358576593000, v:0}, new_max_version={val:1748678358576593000, v:0}, last_min_version={val:1748678358499967000, v:0}, last_max_version={val:1748678358499967000, v:0}, total_time_us=8393, delta_us=367852) [2025-05-31 02:59:18.944502] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678358576593000, v:0}, min_version={val:1748678358576593000, v:0}, max_version={val:1748678358576593000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:18.950178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.950212] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:59:18.952004] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:18.960469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.962085] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:18.971244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.981530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:18.991831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.002207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=146][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.002753] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678358501297], gts:1748678358538589000, latest_srr:[mts=1748678359002718]}) [2025-05-31 02:59:19.012528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.022908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.033212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.043614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.052964] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:19.053075] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=42] ====== check clog disk timer task ====== [2025-05-31 02:59:19.053106] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:19.053881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.064198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.079970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.082635] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=17] gc stale ls task succ [2025-05-31 02:59:19.089353] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:19.090502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.100719] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.102924] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=44] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:19.107309] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83105][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA85-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:19.110232] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=111] PNIO [ratelimit] time: 1748678359110227, bytes: 0, bw: 0.000000 MB/s, add_ts: 1010425, add_bytes: 0 [2025-05-31 02:59:19.110560] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=47] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:19.111724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.116525] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1748678359116522, bytes: 7137173464, bw: 0.254025 MB/s, add_ts: 1003749, add_bytes: 267363 [2025-05-31 02:59:19.118375] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28331][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA3-0-0] [lt=101] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678359152379, keepalive_ts:-1}) [2025-05-31 02:59:19.118425] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28331][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA3-0-0] [lt=46] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678359152379, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:19.124741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=195][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.134237] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665517F563D-0-0] [lt=17] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4121) [2025-05-31 02:59:19.134972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.137772] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5917, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:19.137811] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=396, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:19.137976] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:19.138024] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7623, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:19.138048] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=83, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:19.142246] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=84] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:19.145930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.159004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.162576] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:19.169263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.179649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.191437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=432][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.192466] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3AFE-0-0] [lt=44] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:19.194576] INFO [STORAGE.TRANS] iterate_tx_ctx (ob_ls_tx_service.cpp:487) [81744][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=11] iter set ready success(ret=0, ls_id={id:1}) [2025-05-31 02:59:19.201678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.202786] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=56] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:19.211908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.222159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.234742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.245351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.256305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.266540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.268120] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81744][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=11] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:19.277179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.281275] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1EF0-0-0] [lt=47] handle gts request statistics(total_rt=1417, total_cnt=193, avg_rt=7.304123711340205993e+00, avg_cnt=1.930000000000000071e+01) [2025-05-31 02:59:19.290143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.306679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.308455] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=44] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:19.316893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.328418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.341546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.342079] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:19.342133] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52229, [delay/us]:prepare:11, schedule:270, submit:451, rt:4248, total:4982) [2025-05-31 02:59:19.342152] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:19.342208] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52229, [delay/us]: prepare:11, schedule:270, submit:451, device:4248, total:4982) [2025-05-31 02:59:19.342221] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:19.342247] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4017, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:338, total:354) [2025-05-31 02:59:19.342272] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:19.342375] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=97] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4017, [delay/us]: prepare:0, schedule:0, submit:600000000, device:338, total:354) [2025-05-31 02:59:19.342455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:19.342523] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:19.342535] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:19.342544] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:19.342552] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:19.342560] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=249kB/s, net_out=139kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:19.342599] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399454][2nd=2901]}) [2025-05-31 02:59:19.342938] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=125] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:19.351903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.358807] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.358973] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.359246] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.359884] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.362684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.363627] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=462] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:19.365051] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2025-05-31 02:59:19.372944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.383181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.393522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.405050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.415547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.425778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.436410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.438951] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=37] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:59:19.449415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.459544] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38BB-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525408641712, cache_obj->added_lc()=false, cache_obj->get_object_id()=53692, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:19.459942] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.470182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.480636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.490849] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.501119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.506397] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=112] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:19.506497] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=45] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=2, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:19.513596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=217][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.524450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.535050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.535097] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:19.538175] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [26238][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:19.542354] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] no need to update min/max version(new_version={val:1748678359001631000, v:0}, min_version={val:1748678359001631000, v:0}, max_version={val:1748678359001631000, v:0}) [2025-05-31 02:59:19.543127] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:19.545581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.553493] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=108] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:19.555803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.565944] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:19.566190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.570255] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38BD-0-0] [lt=59][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525408752424, cache_obj->added_lc()=false, cache_obj->get_object_id()=53693, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:19.578090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.588343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.589574] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:19.590428] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.590744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.590784] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.591551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.591609] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.591640] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.595969] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:19.598768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=190][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.603522] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:19.603568] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:19.603677] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:19.603709] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:19.609651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.609772] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:59:19.614275] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] start to print blacklist info [2025-05-31 02:59:19.614760] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] ls blacklist refresh finish(cost_time=37405) [2025-05-31 02:59:19.618617] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [23567][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=40] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:59:19.620166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.632013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.633536] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:341349 disk:44K/10:341349 t1m2:44K/10:341349 t1m2g0:44K/10:341349 |q0: 0 |q1: 0 [2025-05-31 02:59:19.643181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.654360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.656019] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678359502984000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678359655976}, version_delta=153032) [2025-05-31 02:59:19.666789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.674408] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F3E3FB6-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678359669341000, start_id=1748678359669341000, end_id=1748678359669341001) [2025-05-31 02:59:19.677723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.689972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.700365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.713540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.723948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.734251] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.743449] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:19.744301] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:19.745523] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678349737366, end_time=1748678359745519, dag_status={}) [2025-05-31 02:59:19.745559] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745574] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745582] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745597] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745618] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745625] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:19.745635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:59:19.745643] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:59:19.746775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.757225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.766384] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:19.767445] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.770282] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83067][T1_L5_G0][T1][YB42AC108714-000636671ECD5CC7-0-0] [lt=61] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=279, Holding Memory(MB)=134, Tenant Remain Memory(MB)=241, Usable Remain Memory(MB)=144, Last Update Limit Timestamp=1748678359767673, Is Updated=true) [2025-05-31 02:59:19.770318] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83067][T1_L5_G0][T1][YB42AC108714-000636671ECD5CC7-0-0] [lt=33] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.581963014602661133e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=279, Throttle Trigger(MB)=167, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.863002783878612718e+03) [2025-05-31 02:59:19.777980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.788745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.799061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.809798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.821174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.831468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.843486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.854512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.864925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.877360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.880691] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=40] BlockGCTimerTask success(ret=0, cost_time_us=19, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:59:19.883721] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=48] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19333046272, fast_delete_log_mode=false) [2025-05-31 02:59:19.889086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.899351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.905528] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4498-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:19.913560] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:75, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:183950152774, recv_hp_rpc_cnt:74943, recv_np_rpc_cnt:878613, recv_lp_rpc_cnt:0, recv_mysql_cnt:47113, recv_task_cnt:2474, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1233401 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9885, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:544358, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:7, nesting_worker_cnt:0, token_change:1748678359535093}{group_id:10, queue_size:0, recv_req_cnt:104031, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18378, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678359609768}{group_id:2, queue_size:0, recv_req_cnt:151027, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101188, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:108017, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272756, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=692 pcode=0x717:cnt=483 pcode=0x1502:cnt=270 pcode=0x710:cnt=174 pcode=0x523:cnt=100, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:59:19.914605] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1041] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82609854, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:59:19.915347] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=739] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84150565, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:59:19.916023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=666][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.926253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.936581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.944510] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:19.946803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.952268] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=42] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:19.957093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.967056] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:19.967319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.977948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.988775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:19.989100] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=46] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:19.999098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.011971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.022616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.033053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.036751] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=68, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:20.036807] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5199, avg_group_log_size=399) [2025-05-31 02:59:20.036829] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5199) [2025-05-31 02:59:20.036844] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:20.038691] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5199) [2025-05-31 02:59:20.042568] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC120E-0-0] [lt=40] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=4783, avg_log_submit_to_slide_time=5567) [2025-05-31 02:59:20.042928] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5199, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:20.043549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.044190] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=114] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678359843780000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=200397, in_cluster_service=true, cluster_version={val:1748678359502984001, v:0}, min_cluster_version={val:1748678359502984001, v:0}, max_cluster_version={val:1748678359502984001, v:0}, get_cluster_version_err=0, cluster_version_delta=541193, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195441, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=350989419, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:20.050795] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678359780887000, v:0}, new_max_version={val:1748678359780887000, v:0}, last_min_version={val:1748678359502984001, v:0}, last_max_version={val:1748678359502984001, v:0}, total_time_us=6538, delta_us=269904) [2025-05-31 02:59:20.050897] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678359780887000, v:0}, min_version={val:1748678359780887000, v:0}, max_version={val:1748678359780887000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:20.053710] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:20.054365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.064662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.076985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.082751] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-05-31 02:59:20.082947] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=28] gc stale ls task succ [2025-05-31 02:59:20.087219] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.090121] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:20.098922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.103146] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:20.110168] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.110744] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:20.112337] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81740][T1_TimerWK0_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=27] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:20.115211] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1748678360115207, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004980, add_bytes: 0 [2025-05-31 02:59:20.117437] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1748678360117434, bytes: 7137352178, bw: 0.170280 MB/s, add_ts: 1000912, add_bytes: 178714 [2025-05-31 02:59:20.122838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.123672] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA86-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:20.133490] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.145710] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.146429] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:20.156297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.168546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.169626] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:20.178789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.189289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.195318] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83104][T1_L0_G0][T1][YB42AC10870A-000636671538EF58-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1748678360188358000, start_id=1748678360188358001, end_id=1748678360188358002) [2025-05-31 02:59:20.197112] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=94] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=3031, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:20.197178] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1067, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:20.197239] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5077) [2025-05-31 02:59:20.197267] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=38, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:20.198167] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC1210-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:20.198190] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC1210-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=5121, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:20.201435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.212597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.222879] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.233166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.237666] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2075-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:20.238015] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2075-0-0] [lt=339] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=350) [2025-05-31 02:59:20.243440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.253701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.263978] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.274293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.284564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.294802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.309346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.319591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.329993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.335485] INFO [SQL] get_cache_obj (ob_sql_stat_record.cpp:621) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F2E4729-0-0] [lt=24] sql stat record not found(ret=-5138, key={sql_id:"59A18E4B4AE4E8609EFE58284C23F22", plan_hash:3290318591324336132, source_addr:"172.16.135.91:37418"}) [2025-05-31 02:59:20.340440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.344216] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38BA-0-0] [lt=6] rpc statistics(total_trans_msg_count=3, total_batch_msg_count=0) [2025-05-31 02:59:20.346688] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:20.350963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.353518] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:20.353578] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4388, iops:14, norm_iops:3, bw:60745, [delay/us]:prepare:16, schedule:850, submit:244, rt:2364, total:3475) [2025-05-31 02:59:20.353603] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:20.353663] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4389, avg_iops:14, avg_bw:60744, [delay/us]: prepare:16, schedule:850, submit:244, device:2364, total:3475) [2025-05-31 02:59:20.353677] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:20.353703] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4049, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:344, total:357) [2025-05-31 02:59:20.353727] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:20.353756] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4049, [delay/us]: prepare:0, schedule:0, submit:600000000, device:344, total:357) [2025-05-31 02:59:20.353774] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:20.353788] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:20.353797] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:20.353805] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:20.353813] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:20.353822] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=191kB/s, net_out=161kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:20.353843] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399455][2nd=2760]}) [2025-05-31 02:59:20.359062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=137] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.359604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.359644] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.359935] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.361700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.366705] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=40] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:20.366749] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=42] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:20.366918] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4C-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:20.366943] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4C-0-0] [lt=21] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:20.370461] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:20.371181] INFO [COMMON] wash (ob_kvcache_store.cpp:355) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=233] Wash time detail, (compute_wash_size_time=357, refresh_score_time=70, wash_time=0) [2025-05-31 02:59:20.372126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.376524] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=27] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:20.384522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.396243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.398900] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] Garbage Collector is running(seq_=1996, gc_interval=10000000) [2025-05-31 02:59:20.398961] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=32] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:59:20.398993] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=61) [2025-05-31 02:59:20.399306] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=303] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:59:20.399324] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:59:20.399339] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:59:20.399351] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=14) [2025-05-31 02:59:20.399359] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-05-31 02:59:20.399367] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:59:20.399388] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:59:20.407042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.418187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.428552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.441209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.453420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=222][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.464128] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:20.466923] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [30229][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:20.467108] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=279] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:20.469648] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA87-0-0] [lt=51][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525409651814, cache_obj->added_lc()=false, cache_obj->get_object_id()=53695, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:20.476476] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83099][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38BF-0-0] [lt=40][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525409658643, cache_obj->added_lc()=false, cache_obj->get_object_id()=53696, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:20.476799] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:20.481970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.492329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.502664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.508047] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=736, get_gts_with_stc_cnt=61, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=61, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:20.508197] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=147] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678360508015], gts:1748678360507709000, latest_srr:[mts=1748678360508015]}) [2025-05-31 02:59:20.508218] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:20.513611] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.526448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.536729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.546905] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:20.547284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.553405] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=122] ====== checkpoint timer task ====== [2025-05-31 02:59:20.554414] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=48] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:20.554544] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=128] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:20.554852] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:20.555035] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=176] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:20.555102] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=59] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:20.555142] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=35] get rec log scn(service_type_=0, rec_log_ts={val:1748678353762182001, v:0}) [2025-05-31 02:59:20.555154] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:20.555162] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:20.555176] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:20.555195] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:20.555229] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:20.555254] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:20.555278] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:20.555291] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:20.555392] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:20.557732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.569743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.575842] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:20.580197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.590436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.590474] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.590569] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.590609] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.590637] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.590794] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:20.591955] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.592143] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.592326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:20.601153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.611559] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.622158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.632474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.643626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.662571] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC108714-000636671E6D5142-0-0] [lt=108][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525409844739, cache_obj->added_lc()=false, cache_obj->get_object_id()=53697, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:20.663564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.663923] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:64K/14:905041 disk:64K/14:905041 t1m2:64K/14:905041 t1m2g0:64K/14:905041 |q0: 0 |q1: 0 [2025-05-31 02:59:20.673088] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=31][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:20.674695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.685145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.695805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.701747] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DA-0-0] [lt=42] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:20.706509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.711705] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81065][TimerWK3_AutoLSLocRpc][T0][YB42AC1087A6-0006366545EA56E5-0-0] [lt=38] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:59:20.716660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.727058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=151][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.738728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.749158] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=94128, log_service_ts=1748678360651940, min_tx_service_ts=4611686018427387, timestamp={val:1748678360651940000, v:0}) [2025-05-31 02:59:20.749200] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=62, Other_Error_Cnt=0, Submit_Succ_Cnt=38, last_scn="{val:1748678355989106000, v:0}", last_lsn={lsn:2137348622}, last_gts={val:1748678360651940000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:59:20.749462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.749957] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=214] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=456112, replace_num=15728) [2025-05-31 02:59:20.749983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:20.759935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.759843] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:20.760008] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=129] temporary file module use no blocks [2025-05-31 02:59:20.760022] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:20.760038] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:20.760053] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525408940984, flush_io_finished_ret_:0, flush_io_finished_round_:19928, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:20.770839] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.774430] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=36] ObArchiveSender is running(thread_index=0) [2025-05-31 02:59:20.776554] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:20.784180] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.795264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.805775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.810323] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=9] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137388243}, readable_log_tail:{lsn:2137388243}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9964333, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:20.810486] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=161] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:59:20.810522] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81744][T1_TimerWK3_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:59:20.816220] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.823761] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:59:20.823931] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=36] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=52180160, tenant_id=1) [2025-05-31 02:59:20.823971] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=29] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:20.826964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.838513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.850102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.852534] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB4D-0-0] [lt=22] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.629962968826293945e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=281, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.767437218177893101e+03) [2025-05-31 02:59:20.856462] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:20.860822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.871452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.883623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.894035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.904676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.910491] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A4499-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:20.915462] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032536, max_lsn:2137388953, max_scn:{val:1748678360852013000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137388953}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095445913}}, last_submit_lsn:{lsn:2137388365}, last_submit_end_lsn:{lsn:2137388953}, last_submit_log_id:4032536, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137388365}, max_flushed_end_lsn:{lsn:2137388953}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137388953}, last_slide_log_id:4032536, last_slide_scn:{val:1748678360852013000, v:0}, last_slide_lsn:{lsn:2137388365}, last_slide_end_lsn:{lsn:2137388953}, last_slide_log_pid:11, last_slide_log_accum_checksum:2537373496, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032537, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137388953}, readable_log_tail:{lsn:2137388953}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9963623, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:59:20.915898] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=353] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678360855314, last_flushed_end_lsn:{lsn:2137388953}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678360856150, last_flushed_end_lsn:{lsn:2137388953}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678360862173, last_flushed_end_lsn:{lsn:2137388953}}]) [2025-05-31 02:59:20.915955] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=51] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032537, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678360855314, last_flushed_end_lsn:{lsn:2137388953}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678360856150, last_flushed_end_lsn:{lsn:2137388953}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678360862173, last_flushed_end_lsn:{lsn:2137388953}}]) [2025-05-31 02:59:20.916450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.927119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.937584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.947928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.950405] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:20.952582] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=14] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:20.960736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.970923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.978228] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:20.981094] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:20.991337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.001558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.008912] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1748678361008386000, start_id=1748678361008386000, end_id=1748678361008386001) [2025-05-31 02:59:21.009361] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=48] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:21.012296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.012454] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD41-0-0] [lt=8] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678361010914, keepalive_ts:-1}) [2025-05-31 02:59:21.012489] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD41-0-0] [lt=34] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678361010914, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:21.022570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.032878] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.043103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.045973] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=64] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:21.046006] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] try gc retain ctx [2025-05-31 02:59:21.046014] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-31 02:59:21.046116] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678360852013000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678350870242001, v:0}, keep_alive_scn_:{val:1748678355989106000, v:0}, update_ts_:1748678361044964}) [2025-05-31 02:59:21.047074] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:21.047342] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=134] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=118, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:21.047388] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=5686, avg_group_log_size=516) [2025-05-31 02:59:21.047409] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5695) [2025-05-31 02:59:21.047422] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:21.053399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.054765] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=109] get number(ret=0, service_type_=0, range=1, base_id=1748678361053662000, start_id=1748678361053662000, end_id=1748678361053662001) [2025-05-31 02:59:21.055113] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-05-31 02:59:21.055143] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:21.057614] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5686) [2025-05-31 02:59:21.057973] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=145, avg_log_submit_to_flush_time=6085, avg_log_submit_to_slide_time=6414) [2025-05-31 02:59:21.058460] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5686, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:21.059500] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:21.061917] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678360651940000, v:0}, new_max_version={val:1748678360651940000, v:0}, last_min_version={val:1748678360338433000, v:0}, last_max_version={val:1748678360338433000, v:0}, total_time_us=9406, delta_us=409974) [2025-05-31 02:59:21.061957] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678360651940000, v:0}, min_version={val:1748678360651940000, v:0}, max_version={val:1748678360651940000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:21.066504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.066701] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=48] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:59:21.076667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.083002] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=56] current monitor number(seq_=-1) [2025-05-31 02:59:21.084672] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=48] gc stale ls task succ [2025-05-31 02:59:21.089533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.090994] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:21.099822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.104005] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:21.111058] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:21.112110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.115924] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=69] PNIO [ratelimit] time: 1748678361115920, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000713, add_bytes: 0 [2025-05-31 02:59:21.120469] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81482][EvtHisUpdTask][T1][Y0-0000000000000000-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1748678361117070000, start_id=1748678361117070000, end_id=1748678361117070001) [2025-05-31 02:59:21.121422] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1748678361121419, bytes: 7137512921, bw: 0.152688 MB/s, add_ts: 1003985, add_bytes: 160743 [2025-05-31 02:59:21.122654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.131084] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672F2E472C-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678361130967000, start_id=1748678361130967000, end_id=1748678361130967001) [2025-05-31 02:59:21.131399] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83088][T1_L0_G0][T1][YB42AC1087A8-000636672F2E472C-0-0] [lt=40] transaction rpc statistics(total_rt=3034, total_process=12, avg_rt=233) [2025-05-31 02:59:21.132870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.143307] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.148607] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA87-0-0] [lt=51][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:21.150292] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137390488}], scn_array=[cnt:1, {val:1748678361130967000, v:0}], curr_size=1002, accum_size=2424, time_cost=18725) [2025-05-31 02:59:21.150915] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:21.153995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.164227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.175767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.176239] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=30] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:21.176272] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:21.176309] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=32] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=960) [2025-05-31 02:59:21.181099] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:21.186224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=158][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.199016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.209342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.219752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.230166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=163][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.230755] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=58] === Run print tenant memory usage task === [2025-05-31 02:59:21.230818] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 121,196,544 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 647,544,832 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,361,229,784 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 52,180,160 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 902,164,480 max_mem_memstore_can_get_now= 260,900,826 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:59:21.231349] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=87] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,073,150,976 total_hold= 3,323,985,920 used= 1,580,879,872 freelists_hold= 1,492,271,104 total_maps= 2,019 total_unmaps= 807 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 121,196,544 virtual_memory_used= 4,344,737,792 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 998,244,352 count= 476 pushes= 14,531,630 pops= 14,531,154 maps= 1,824 unmaps= 763 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,950 pops= 22,946 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:59:21.231380] INFO print (ob_malloc_time_monitor.cpp:37) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 4870, delta_count= 1, avg_cost_time= 4870 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:59:21.241231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.250672] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT WRITE LOG COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=5702, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:21.250735] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=1426, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:21.251562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.252644] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665517F5657-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=7521) [2025-05-31 02:59:21.258154] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:21.258195] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=8421, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:21.258219] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=74, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:21.261882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.272250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.282677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.296140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.309186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.321455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.331682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.341970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.351659] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:21.352360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=157][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.358547] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=69] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.358960] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.360529] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.362505] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.363048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.368658] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:21.368713] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, norm_iops:3, bw:52453, [delay/us]:prepare:14, schedule:987, submit:559, rt:3947, total:5508) [2025-05-31 02:59:21.368732] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:21.368791] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:52454, [delay/us]: prepare:14, schedule:987, submit:559, device:3947, total:5508) [2025-05-31 02:59:21.368805] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:21.368946] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4034, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:420, total:437) [2025-05-31 02:59:21.368980] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:21.369015] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4034, [delay/us]: prepare:0, schedule:0, submit:600000000, device:420, total:437) [2025-05-31 02:59:21.369034] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:21.369049] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:21.369070] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:21.369097] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:21.369106] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:21.369125] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SERVER] net_in=141kB/s, net_out=99kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:21.369172] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] successfully alloc block(macro_id={[ver=1,mode=0,seq=399456][2nd=2914]}) [2025-05-31 02:59:21.373457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.381726] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:21.383267] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=60] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2025-05-31 02:59:21.383935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.387042] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=41] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:21.394586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.404940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.416038] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.426612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.437005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.448337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=200][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.456247] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=114] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:21.457944] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=58] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:21.458057] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=31] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:21.458096] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=37] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678352212516000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678352212516000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:59:21.458791] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [30229][T1_L0_G1][T1][YB42AC10871E-0006366549CC121F-0-0] [lt=348] init point thread id with(&point=0x55c0908df900, idx_=1557, point=[thread id=30229, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=30229) [2025-05-31 02:59:21.459308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.463377] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137392201}], scn_array=[cnt:1, {val:1748678361254262001, v:0}], curr_size=122, accum_size=3257, time_cost=12856) [2025-05-31 02:59:21.471355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.482527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.482582] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:21.482689] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [28540][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=65] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:21.486846] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83085][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38C1-0-0] [lt=52][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525410669017, cache_obj->added_lc()=false, cache_obj->get_object_id()=53698, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:21.492630] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=10] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:21.493347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:21.494350] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [30576][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 03:59:21.494404] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [30576][T1_L0_G19][T1][YB42AC10871E-00063665511850DA-0-0] [lt=19] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 03:59:21.494429] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [30576][T1_L0_G19][T1][YB42AC10871E-00063665511850DA-0-0] [lt=16] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:21.494598] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:59:21.497904] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=49] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:21.500148] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=29] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:21.502264] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=20] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:21.502433] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=167] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:21.504284] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:21.504354] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=19] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:21.504369] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:21.504718] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=334] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:21.505357] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=633] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:21.505403] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=35] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:21.505416] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=10] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:21.505439] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=20] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:21.505460] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:21.506795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.507777] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=7] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:21.507799] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=21] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:21.507809] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:21.510024] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] ts response task statistics(alloc_count=486818, free_count=243408) [2025-05-31 02:59:21.510167] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=74] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678361008978], gts:1748678361047724000, latest_srr:[mts=1748678361510130]}) [2025-05-31 02:59:21.512534] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:21.512563] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=28] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:21.512575] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=10] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:21.512589] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=10] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:21.512607] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BDF-0-0] [lt=16] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:21.517753] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.528065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.539940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.550228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.552172] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:21.568552] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:21.568650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.584303] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:21.588228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.594250] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83088][T1_L0_G0][T1][YB42AC108714-000636671E2D3A74-0-0] [lt=17] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:21.595721] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.595763] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598679] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598729] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598812] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:21.598973] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.599177] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=51] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:21.600336] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=44][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b097cd45aa0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096499} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:24547} write_epoch=1, prev=this=0x2b095e50aba8 trans_version={val:1748676671500096000, v:0} scn={val:1748676671500096000, v:0} tx_id={txid:10094547} prev=0x2b095e50a9f0 next=(nil) modify_count=7 acc_checksum=1074106567 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678361504002000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:37844} write_epoch=3, snapshot_version={val:1748678361591963000, v:0}, *this={this=0x2b09b3163990 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e50aba8 latest_compact_node=0x2b09802dd100 max_trans_version={val:1748678301285689000, v:0} max_trans_id=10096434 max_elr_trans_version={val:1748678301285689000, v:0} max_elr_trans_id=10096434 latest_compact_ts=1748675524407946 last_compact_cnt=5 total_trans_node_cnt=8 max_modify_scn={val:1748676671500096000, v:0} min_modify_scn={val:1748674981271724000, v:0}}) [2025-05-31 02:59:21.600456] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=117][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b097cd45aa0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096499} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:24547} write_epoch=1, ctx={this:0x2b091c9511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678391489878, tablet_id:{id:309}, table_iter:0x2b091c94f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678391489878, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678361591963000, v:0}, tx_id:{txid:10096499}, scn:{branch:0, seq:14179}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10096499}, tx_desc:{this:0x2b095dbf8640, tx_id:{txid:10096499}, state:1, addr:"172.16.135.167:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678361600001, commit_ts:-1, finish_ts:-1, timeout_us:29999983, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678361609097, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cb39d0, mem_ctx:0x2b0904cb4f50, tx_scn:{branch:0, seq:24547}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:21.600557] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10096499}, data:0x2b091c94e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:24547}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:21.600575] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.167", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10096499} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:21.600606] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.600621] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099cc0c820, local_datums:0x2b099cc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b099cc0c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b099cc0c4e2, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c4f2, hex: 169C91E869360600, int: 1748678361586710}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c4fa, hex: 169C91E869360600, int: 1748678361586710}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c502, hex: BCA0F20700000000, int: 133341372}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099cc0c50a, hex: 0040F00900000000, int: 166739968}]}) [2025-05-31 02:59:21.600790] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.605541] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=53] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:29763}, to_seq_no={branch:0, seq:14180}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096499} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.605620] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83059][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD6-0-0] [lt=40] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-05-31 02:59:21.609435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.616750] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:164) [81740][T1_TimerWK0_MediumLoop][T1][YB42AC1087A6-00063665444A5F41-0-0] [lt=14] MediumCheckTask(cost_ts=10) [2025-05-31 02:59:21.623935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.633229] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDD8-0-0] [lt=51][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:21.633281] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDD8-0-0] [lt=46][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.633302] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDD8-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b092360c4e2, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f4, hex: 9C2B92E869360600, int: 1748678361623452}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4fc, hex: 9C2B92E869360600, int: 1748678361623452}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c504, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c50c, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:21.633468] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDD8-0-0] [lt=69][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.634698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=268][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.635190] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDD8-0-0] [lt=32] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6707}, to_seq_no={branch:0, seq:6702}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096500} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.644967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.655944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.664002] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:64K/14:723191 disk:64K/14:723191 t1m2:64K/14:723191 t1m2g0:64K/14:723191 |q0: 0 |q1: 0 [2025-05-31 02:59:21.666522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:21.667340] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [30620][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:21.668819] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:21.669772] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=13][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af2040 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096501} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:18080} write_epoch=1, prev=this=0x2b0988ae4058 trans_version={val:1748678301345729000, v:0} scn={val:1748678301345729000, v:0} tx_id={txid:10096436} prev=0x2b097cd45220 next=(nil) modify_count=115 acc_checksum=1601431375 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678361504002000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:22714} write_epoch=3, snapshot_version={val:1748678361662046001, v:0}, *this={this=0x2b09b3164210 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae4058 latest_compact_node=0x2b0980345740 max_trans_version={val:1748678301345729000, v:0} max_trans_id=10096436 max_elr_trans_version={val:1748678301345729000, v:0} max_elr_trans_id=10096436 latest_compact_ts=1748678241025171 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678301345729000, v:0} min_modify_scn={val:1748674860018513001, v:0}}) [2025-05-31 02:59:21.669957] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=60][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af2040 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096501} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:18080} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678391560314, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678391560314, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678361662046001, v:0}, tx_id:{txid:10096501}, scn:{branch:0, seq:13051}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10096501}, tx_desc:{this:0x2b094e869c60, tx_id:{txid:10096501}, state:1, addr:"172.16.135.167:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678361668512, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678361685812, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:18080}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:21.670068] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=110][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10096501}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:18080}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:21.670087] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.167", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10096501} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:21.670119] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.670134] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b092360c4e2, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f2, hex: A4AD92E869360600, int: 1748678361656740}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4fa, hex: A4AD92E869360600, int: 1748678361656740}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c502, hex: C20B686300000000, int: 1667763138}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c50a, hex: C20B686300000000, int: 1667763138}]}) [2025-05-31 02:59:21.670245] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.674821] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83082][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDA-0-0] [lt=32] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:18083}, to_seq_no={branch:0, seq:13052}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096501} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.677594] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81743][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=21] ====== traversal_flush timer task ====== [2025-05-31 02:59:21.678009] INFO [MDS] flush (mds_table_mgr.cpp:184) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF3-0-0] [lt=24] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678361657315000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678370364653000, latest_log_ts:{val:1748678353762182001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:21.678207] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF3-0-0] [lt=190] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:21.680708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=571][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.689244] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDB-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678361688226000, start_id=1748678361688226000, end_id=1748678361688226001) [2025-05-31 02:59:21.691603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.702309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.708479] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83101][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDD-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678361707313000, start_id=1748678361707313000, end_id=1748678361707313001) [2025-05-31 02:59:21.712613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:21.719602] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [30620][T1_L0_G1][T1][YB42AC1087A7-00063665517F5660-0-0] [lt=25] init point thread id with(&point=0x55c0908e5b00, idx_=1949, point=[thread id=30620, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=30620) [2025-05-31 02:59:21.723314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.724307] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=68][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af2668 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096503} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:6243} write_epoch=1, prev=this=0x2b095e50b620 trans_version={val:1748676671632015000, v:0} scn={val:1748676671632015000, v:0} tx_id={txid:10094551} prev=0x2b095e50b468 next=(nil) modify_count=7 acc_checksum=3406705100 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678361504002000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:18} write_epoch=3, snapshot_version={val:1748678361720814000, v:0}, *this={this=0x2b09b31654a0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b095e50b620 latest_compact_node=0x2b09802dd540 max_trans_version={val:1748678301401236000, v:0} max_trans_id=10096438 max_elr_trans_version={val:1748678301401236000, v:0} max_elr_trans_id=10096438 latest_compact_ts=1748675524527156 last_compact_cnt=5 total_trans_node_cnt=8 max_modify_scn={val:1748676671632015000, v:0} min_modify_scn={val:1748674981418012000, v:0}}) [2025-05-31 02:59:21.724365] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=55][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af2668 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096503} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:6243} write_epoch=1, ctx={this:0x2b091cbcf1a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678391617483, tablet_id:{id:309}, table_iter:0x2b091cbcd6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678391617483, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678361720814000, v:0}, tx_id:{txid:10096503}, scn:{branch:0, seq:4579}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10096503}, tx_desc:{this:0x2b0991239360, tx_id:{txid:10096503}, state:1, addr:"172.16.135.167:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678361723413, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678361746826, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:6243}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:21.724458] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=91][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10096503}, data:0x2b091cbcc0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:6243}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:21.724476] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.167", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10096503} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:21.724571] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=92][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.724609] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=30][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099e20c820, local_datums:0x2b099e20c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b099e20c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b099e20c4e2, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c4f3, hex: CC8A93E869360600, int: 1748678361713356}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c4fb, hex: CC8A93E869360600, int: 1748678361713356}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c503, hex: 00C0C30A00000000, int: 180600832}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099e20c50b, hex: 00C0C30A00000000, int: 180600832}]}) [2025-05-31 02:59:21.724905] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83095][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=87][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.727738] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDE-0-0] [lt=15] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6246}, to_seq_no={branch:0, seq:4580}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096503} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.734004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.742570] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83106][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDDF-0-0] [lt=54] get number(ret=0, service_type_=0, range=1, base_id=1748678361741025000, start_id=1748678361741025000, end_id=1748678361741025001) [2025-05-31 02:59:21.744566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=287][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.745035] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=114] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=57153572, this={dio_aligned_buf:{buf_write_offset:2647, buf_padding_size:1449, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:390579, truncate_used_ts:132330}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=57151488, buf_len=563, total_write_size=67888, total_write_size_after_dio=634880, ob_pwrite_used_ts=565623, count=139, each_pwrite_cost=4069) [2025-05-31 02:59:21.755516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.755637] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:21.763679] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=11] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=70) [2025-05-31 02:59:21.763811] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=31][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af28e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096504} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:15109} write_epoch=1, prev=this=0x2b097ccebca0 trans_version={val:1748675524563263000, v:0} scn={val:1748675524563263000, v:0} tx_id={txid:10093229} prev=0x2b09b31fda50 next=(nil) modify_count=3 acc_checksum=3765322303 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678361504002000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:25117} write_epoch=3, snapshot_version={val:1748678361755694000, v:0}, *this={this=0x2b09b3165740 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b097ccebca0 latest_compact_node=(nil) max_trans_version={val:1748678301439289000, v:0} max_trans_id=10096439 max_elr_trans_version={val:1748678301439289000, v:0} max_elr_trans_id=10096439 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=4 max_modify_scn={val:1748675524563263000, v:0} min_modify_scn={val:1748674981458313000, v:0}}) [2025-05-31 02:59:21.763960] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=147][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af28e8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096504} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:15109} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678391657569, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678391657569, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678361755694000, v:0}, tx_id:{txid:10096504}, scn:{branch:0, seq:7906}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10096504}, tx_desc:{this:0x2b094e9fc700, tx_id:{txid:10096504}, state:1, addr:"172.16.135.167:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678361760437, commit_ts:-1, finish_ts:-1, timeout_us:29999980, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678361778937, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:15109}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:21.764056] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10096504}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:15109}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:21.764075] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.167", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10096504} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:21.764105] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.764120] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=7][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b092360c4e2, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f2, hex: 281D94E869360600, int: 1748678361750824}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4fa, hex: 281D94E869360600, int: 1748678361750824}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c502, hex: 00E0430000000000, int: 4448256}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c50a, hex: 00E0430000000000, int: 4448256}]}) [2025-05-31 02:59:21.764222] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.764603] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:21.767031] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1748678361764527000, start_id=1748678361764527000, end_id=1748678361764527001) [2025-05-31 02:59:21.768641] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.773641] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83106][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDE0-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:25908}, to_seq_no={branch:0, seq:7907}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096504} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.779594] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.785012] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44412928, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:21.790090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.791953] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=20] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:21.791999] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=38] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:21.792027] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=24] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:21.792048] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:21.796312] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] Dumping All Cache Objs(alloc_obj_list.count()=314, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53694, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:21.797290] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=971] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:21.798136] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC10871E-000636654DFA0E15-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1748678361797787000, start_id=1748678361797787000, end_id=1748678361797787001) [2025-05-31 02:59:21.800624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.801585] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137400232}], scn_array=[cnt:1, {val:1748678361787085000, v:0}], curr_size=561, accum_size=11727, time_cost=10999) [2025-05-31 02:59:21.811308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.814038] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDE3-0-0] [lt=62] get number(ret=0, service_type_=0, range=1, base_id=1748678361810923000, start_id=1748678361810923000, end_id=1748678361810923001) [2025-05-31 02:59:21.822403] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.827779] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af2ab8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096506} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3143} write_epoch=1, prev=this=0x2b098030d6f8 trans_version={val:1748676671765840000, v:0} scn={val:1748676671765840000, v:0} tx_id={txid:10094554} prev=0x2b095e50be18 next=(nil) modify_count=5 acc_checksum=417576174 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678361504002000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:33904} write_epoch=0, snapshot_version={val:1748678361822072000, v:0}, *this={this=0x2b097ccd0290 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b098030d6f8 latest_compact_node=0x2b098030d6f8 max_trans_version={val:1748678301482137000, v:0} max_trans_id=10096441 max_elr_trans_version={val:1748678301482137000, v:0} max_elr_trans_id=10096441 latest_compact_ts=1748676671776278 last_compact_cnt=5 total_trans_node_cnt=6 max_modify_scn={val:1748676671765840000, v:0} min_modify_scn={val:1748674981539922000, v:0}}) [2025-05-31 02:59:21.828017] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=180][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af2ab8 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:10096506} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3143} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678391721642, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678391721642, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678361822072000, v:0}, tx_id:{txid:10096506}, scn:{branch:0, seq:3140}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:10096506}, tx_desc:{this:0x2b094e80cb20, tx_id:{txid:10096506}, state:1, addr:"172.16.135.167:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678361826390, commit_ts:-1, finish_ts:-1, timeout_us:29999981, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678361850683, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:3143}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:21.828130] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=112][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:10096506}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3143}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:21.828149] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"172.16.135.167", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":2},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:10096506} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:21.828320] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=167][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.828341] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b092360c4e2, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f2, hex: 392395E869360600, int: 1748678361817913}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4fa, hex: 392395E869360600, int: 1748678361817913}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c502, hex: 0080D30000000000, int: 13860864}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c50a, hex: 0080D30000000000, int: 13860864}]}) [2025-05-31 02:59:21.828489] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=86][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.831281] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDE4-0-0] [lt=73] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:3146}, to_seq_no={branch:0, seq:3141}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096506} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.832950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.843207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.853313] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137401385}], scn_array=[cnt:1, {val:1748678361841423000, v:0}], curr_size=561, accum_size=12880, time_cost=10794) [2025-05-31 02:59:21.853957] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.864818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.875192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.886679] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE8-0-0] [lt=35][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:21.886748] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE8-0-0] [lt=64][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:21.886770] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE8-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b092360c820, local_datums:0x2b092360c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x2b092360c4c4, hex: 3137322E31362E3133352E313637, cstr: 172.16.135.167}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4d2, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4da, hex: 0200000000000000, int: 2}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b092360c4e2, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f1, hex: 940296E869360600, int: 1748678361875092}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c4f9, hex: 940296E869360600, int: 1748678361875092}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b092360c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:21.886973] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3F0-0-0] [lt=84][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525411069142, cache_obj->added_lc()=false, cache_obj->get_object_id()=53699, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:21.886987] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC1087A7-00063665445DFDE8-0-0] [lt=121][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:21.888077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.888681] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC1087A7-00063665445DFDE8-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:2601}, to_seq_no={branch:0, seq:2596}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:10096508} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:21.898663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.909415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.913371] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449A-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:21.919664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.930399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.941981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.952796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.953421] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:21.955824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:21.963540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.976165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.985484] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:21.986529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:21.996945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.009485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.010994] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:22.011407] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=64] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678361510453], gts:1748678361549421000, latest_srr:[mts=1748678362011379]}) [2025-05-31 02:59:22.020184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.030459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.040789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.051983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.057477] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=112, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:22.057554] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=24, avg_log_batch_cnt=1, total_group_log_size=14736, avg_group_log_size=614) [2025-05-31 02:59:22.057577] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=14727) [2025-05-31 02:59:22.057594] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:22.060789] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=67] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=14736) [2025-05-31 02:59:22.060952] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=24, avg_log_gen_to_freeze_time=303, avg_log_gen_to_submit_time=443, avg_log_submit_to_flush_time=6329, avg_log_submit_to_slide_time=6977) [2025-05-31 02:59:22.061124] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=14736, hit_cnt=25, read_cnt=25, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:22.062470] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB59-0-0] [lt=15] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.606417655944824219e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=280, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.813808123964660808e+03) [2025-05-31 02:59:22.065160] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.068212] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678361810923000, v:0}, new_max_version={val:1748678361810923000, v:0}, last_min_version={val:1748678361648673000, v:0}, last_max_version={val:1748678361648673000, v:0}, total_time_us=8163, delta_us=257286) [2025-05-31 02:59:22.068251] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678361810923000, v:0}, min_version={val:1748678361810923000, v:0}, max_version={val:1748678361810923000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:22.068274] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:22.071697] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:22.075720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.083950] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:59:22.084811] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=27] gc stale ls task succ [2025-05-31 02:59:22.089161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.099380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.101086] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:22.106106] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:22.110602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.111135] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:22.121106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.122613] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1748678362122585, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006665, add_bytes: 0 [2025-05-31 02:59:22.126171] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1748678362126168, bytes: 7137880091, bw: 0.348506 MB/s, add_ts: 1004749, add_bytes: 367170 [2025-05-31 02:59:22.133359] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B00-0-0] [lt=46] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:22.133705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.144272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.154945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.156094] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:22.164172] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC10870A-0006366715B99E26-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678362163858000, start_id=1748678362163858000, end_id=1748678362163858001) [2025-05-31 02:59:22.164522] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81431][pnio1][T0][YB42AC10870A-000636671AB9479B-0-0] [lt=10][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:189319743, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678362194739, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1389230, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678362155748, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678362164518, enqueue_timestamp_:0, request_arrival_time_:1748678362164518, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:22.164587] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81431][pnio1][T0][YB42AC10870A-000636671AB9479B-0-0] [lt=50][errcode=-5150] deliver rpc request fail(&req=0x2b0959610098, ret=-5150) [2025-05-31 02:59:22.165179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.177787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.186406] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40218624, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:22.186704] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA88-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:22.188159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.198685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.210687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.221446] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.236237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.243024] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2076-0-0] [lt=92] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:22.243061] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2076-0-0] [lt=29] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=39) [2025-05-31 02:59:22.248572] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.250785] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=4733, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:22.250821] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=23, stat_interval=1000000, avg (count/cost)=572, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:22.257187] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665517F566E-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=14233) [2025-05-31 02:59:22.258790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.269811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.272243] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=10, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:22.272293] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT LOG LIFE TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=6389, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:22.272319] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=24, stat_interval=1000000, avg (count/cost)=61, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:22.282172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.294352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.308939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.322581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.332659] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC108714-000636671EBE4269-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1748678362332570000, start_id=1748678362332570000, end_id=1748678362332570001) [2025-05-31 02:59:22.333398] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.343654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.355327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.356526] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:22.358121] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.358594] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.358943] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.358986] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.365628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.370466] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=109] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:22.370534] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=67] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:22.370587] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4D-0-0] [lt=64] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:22.370608] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4D-0-0] [lt=17] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:22.372607] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:22.372678] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4630, iops:23, norm_iops:6, bw:106075, [delay/us]:prepare:12, schedule:389, submit:407, rt:3758, total:4568) [2025-05-31 02:59:22.373026] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=337] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=6, iops=6, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:22.373137] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4630, avg_iops:23, avg_bw:106035, [delay/us]: prepare:12, schedule:389, submit:407, device:3758, total:4568) [2025-05-31 02:59:22.373154] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:22.373231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4078, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:449, total:465) [2025-05-31 02:59:22.373259] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:22.373344] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4078, [delay/us]: prepare:0, schedule:0, submit:600000000, device:449, total:465) [2025-05-31 02:59:22.373366] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:22.373381] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:22.373390] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:22.373398] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:22.373456] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:22.373469] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SERVER] net_in=410kB/s, net_out=179kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:22.373490] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399457][2nd=2491]}) [2025-05-31 02:59:22.384708] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83097][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33D7-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525411566802, cache_obj->added_lc()=false, cache_obj->get_object_id()=53700, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.385085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.388461] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40218624, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:22.398308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.408547] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.420315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.431211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.441439] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.451923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.464416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.474594] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.478137] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:22.484983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.494964] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38C3-0-0] [lt=39][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525411677132, cache_obj->added_lc()=false, cache_obj->get_object_id()=53701, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.495225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.495296] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:59:22.499220] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [29577][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=31] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:59:22.508127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.519097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.531414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=265][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.541652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.551903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.558615] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:22.562983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.572987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:22.573236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.583436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.590637] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.590701] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.590703] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.590731] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.591469] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-29732864, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:22.591547] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.591954] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.593707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.594953] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:22.595965] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83106][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38C5-0-0] [lt=106][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525411778136, cache_obj->added_lc()=false, cache_obj->get_object_id()=53702, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.597601] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83086][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38C7-0-0] [lt=52][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:22.601440] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:22.602032] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81065][TimerWK3_ServerTracerTim][T0][YB42AC1087A6-0006366545EA56EA-0-0] [lt=13] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:22.602549] WDIAG get_refactored (ob_hashtable.h:1120) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38C9-0-0] [lt=69][errcode=-4201] hashtable not init [2025-05-31 02:59:22.604629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.613281] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:22.613323] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.613561] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.613596] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:22.615095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=344][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.616648] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9] ls blacklist refresh finish(cost_time=36945) [2025-05-31 02:59:22.626116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.636856] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.647090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.657329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.668944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.669056] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:22.669170] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [30229][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=36] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:22.679226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.689693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.699891] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.709019] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DB-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:22.711099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.721317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.731629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.742421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.755356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.760845] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:22.761071] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:22.761117] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] temporary file module use no blocks [2025-05-31 02:59:22.761130] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:22.761146] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:22.761161] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525410942569, flush_io_finished_ret_:0, flush_io_finished_round_:19930, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:22.768007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.769753] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [30576][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1B0-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:22.770140] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [30576][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1B0-0-0] [lt=356] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:22.778620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.788811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.794046] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:22.799282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.805829] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=44] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:96K/21:357957 disk:96K/21:357957 t1m2:96K/21:357957 t1m2g0:96K/21:357957 |q0: 0 |q1: 0 [2025-05-31 02:59:22.810044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.820247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.830614] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [83067][T1_L5_G0][T1][YB42AC1087A8-00063667388E274A-0-0] [lt=44] need check schema(new_schema_version=1748674258961328, cached_tenant_schema_version=1748674258961328, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT primary_zone FROM __all_tenant where tenant_id = ?) [2025-05-31 02:59:22.833203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.839066] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-05-31 02:59:22.839100] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=25] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=51970440, tenant_id=1) [2025-05-31 02:59:22.839123] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:22.845243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.855761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.868557] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.879042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.880546] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1748678362668703000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137408888}, palf_committed_end_scn_:{val:1748678362872247000, v:0}, last_check_scn_:{val:1748678362872247000, v:0}, max_applied_cb_scn_:{val:1748678362668703000, v:0}}) [2025-05-31 02:59:22.880678] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137408888}, readable_log_tail:{lsn:2137408888}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9943688, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:22.889965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=561][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.900301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.905448] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=27] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:22.910537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.916033] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449B-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:22.920794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.931316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.945213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.951585] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:22.951623] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=36] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:22.952267] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=64] update_check_range(min_check_version=1985, max_check_version=1986, global_version=1987) [2025-05-31 02:59:22.952806] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:22.961956] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=35] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:22.962697] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=471840, replace_num=15728) [2025-05-31 02:59:22.962719] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:22.963308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.973963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:22.979729] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 155 275 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 52 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 18 7 3 0 1 20 2 1 0 500 0 88 252 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 2 1 0 500 16 1 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 27484 [2025-05-31 02:59:22.979783] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 904,261,632 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 459,644,928 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 109,051,904 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:22.980259] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 459,644,928 used= 348,622,080 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 24,559,616 used= 24,336,960 count= 34 avg_used= 715,792 block_cnt= 34 chunk_cnt= 22 mod=MysqlRequesReco [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,274,496 used= 6,389,760 count= 108 avg_used= 59,164 block_cnt= 108 chunk_cnt= 29 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,850,816 used= 2,408,160 count= 87 avg_used= 27,680 block_cnt= 87 chunk_cnt= 25 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,015,808 used= 498,976 count= 62 avg_used= 8,048 block_cnt= 62 chunk_cnt= 21 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 416,768 used= 399,872 count= 88 avg_used= 4,544 block_cnt= 88 chunk_cnt= 24 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 286,528 used= 275,824 count= 5 avg_used= 55,164 block_cnt= 5 chunk_cnt= 3 mod=CommSysVarFac [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 100,768 used= 26,128 count= 370 avg_used= 70 block_cnt= 100 chunk_cnt= 27 mod=Coro [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 57,024 used= 44,352 count= 66 avg_used= 672 block_cnt= 61 chunk_cnt= 23 mod=[T]ObWarningBuf [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 48,000 used= 33,600 count= 75 avg_used= 448 block_cnt= 44 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 45,408 used= 4,824 count= 201 avg_used= 24 block_cnt= 76 chunk_cnt= 27 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=SqlSession [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PlanVaIdx [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,656 used= 3,072 count= 8 avg_used= 384 block_cnt= 8 chunk_cnt= 7 mod=TableLockNode [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceMap [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 348,622,080 used= 340,539,323 count= 32,611 avg_used= 10,442 mod=SUMMARY [2025-05-31 02:59:22.980410] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=81] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:59:22.980434] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:59:22.980454] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,595,328 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,528,064 used= 1,491,968 count= 188 avg_used= 7,936 block_cnt= 188 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,364,352 used= 1,243,968 count= 627 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,595,328 used= 35,261,289 count= 3,361 avg_used= 10,491 mod=SUMMARY [2025-05-31 02:59:22.980472] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:59:22.980484] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 109,051,904 used= 105,283,584 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 block_cnt= 204 chunk_cnt= 52 mod=CoStack [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:59:22.980511] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:59:22.980553] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:59:22.980570] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:22.980582] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:22.980623] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:59:22.980649] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 9,042,816 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,852,160 used= 5,713,920 count= 720 avg_used= 7,936 block_cnt= 720 chunk_cnt= 5 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 9,042,816 used= 8,875,584 count= 724 avg_used= 12,259 mod=SUMMARY [2025-05-31 02:59:22.980666] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:22.980700] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 647,544,832 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:22.981101] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 180,835,824 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:23.081044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.081821] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=226] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678362872247000, v:0}, new_max_version={val:1748678362872247000, v:0}, last_min_version={val:1748678362601406000, v:0}, last_max_version={val:1748678362601406000, v:0}, total_time_us=7884, delta_us=209570) [2025-05-31 02:59:23.081940] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=112] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678362872247000, v:0}, min_version={val:1748678362872247000, v:0}, max_version={val:1748678362872247000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:23.081972] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:23.087497] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:59:23.091262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.101982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.102180] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:23.106320] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=43] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:23.111322] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:23.112549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.122787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.127966] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=39] PNIO [ratelimit] time: 1748678363127962, bytes: 7138131526, bw: 0.239358 MB/s, add_ts: 1001794, add_bytes: 251435 [2025-05-31 02:59:23.130048] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=51] PNIO [ratelimit] time: 1748678363130044, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007459, add_bytes: 0 [2025-05-31 02:59:23.130679] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28331][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA4-0-0] [lt=32] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678363165196, keepalive_ts:-1}) [2025-05-31 02:59:23.130752] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28331][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA4-0-0] [lt=71] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678363165196, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:23.134002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.144257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.154521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.162925] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:23.164980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.177709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.187966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.195421] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:23.199405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.206275] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA89-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:23.209703] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.221923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.232346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.242639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.253114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.264316] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3171, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:23.264360] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=414, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:23.264424] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4424) [2025-05-31 02:59:23.267364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.278538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.279957] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:23.280200] INFO [LIB] stat (utility.h:1211) [29190][T1_L0_G1][T1][YB42AC1087A7-00063665517F567A-0-0] [lt=55] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:23.280224] INFO [LIB] stat (utility.h:1211) [29190][T1_L0_G1][T1][YB42AC1087A7-00063665517F567A-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4681, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:23.292500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.302772] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.313766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=213][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.327158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.337452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.347916] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.358477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.359113] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.359155] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.359581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.359809] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=143] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.367059] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:23.369931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.377542] INFO pn_ratelimit (group.c:653) [81060][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:23.380154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.380816] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:23.380945] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=90] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48749, [delay/us]:prepare:14, schedule:359, submit:94, rt:1805, total:2273) [2025-05-31 02:59:23.380965] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:23.381092] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=109] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48764, [delay/us]: prepare:14, schedule:359, submit:94, device:1805, total:2273) [2025-05-31 02:59:23.381108] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:23.381135] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4063, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:395, total:409) [2025-05-31 02:59:23.381160] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:23.381212] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4064, [delay/us]: prepare:0, schedule:0, submit:600000000, device:395, total:409) [2025-05-31 02:59:23.381233] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:23.381270] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:23.381280] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:23.381288] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:23.381296] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:23.381304] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=193kB/s, net_out=148kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:23.381344] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] successfully alloc block(macro_id={[ver=1,mode=0,seq=399458][2nd=2773]}) [2025-05-31 02:59:23.390496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.396044] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:23.397591] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=91] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2025-05-31 02:59:23.400799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.411992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.422206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.432542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.443095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.455946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.469714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.482136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:23.487286] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [31227][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:23.487346] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:23.499121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.503343] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38CB-0-0] [lt=88][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525412685512, cache_obj->added_lc()=false, cache_obj->get_object_id()=53703, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:23.510021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.516587] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=48] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678363516564], gts:1748678363515365000, latest_srr:[mts=1748678363516564]}) [2025-05-31 02:59:23.522757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.535457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=184][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.548151] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.556965] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:23.558387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.567263] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:23.570975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.577570] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:23.581170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.590786] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.590839] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.590932] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.593288] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38CD-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525412775460, cache_obj->added_lc()=false, cache_obj->get_object_id()=53704, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:23.593969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.594099] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.594255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.594309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.594356] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:23.598057] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:23.603363] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=54] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:23.604208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.614505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.627674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.644965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.655222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.665443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.679166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=183][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.679418] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] no need to update min/max version(new_version={val:1748678363274082000, v:0}, min_version={val:1748678363274082000, v:0}, max_version={val:1748678363274082000, v:0}) [2025-05-31 02:59:23.685699] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=29] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:23.685726] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=14] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=987) [2025-05-31 02:59:23.690402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.700650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.710949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.721868] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.736700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=191][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.747080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.765237] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A48-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525412947406, cache_obj->added_lc()=false, cache_obj->get_object_id()=53705, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:23.765773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.766419] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=99] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:360418 disk:48K/11:360418 t1m2:48K/11:360418 t1m2g0:48K/11:360418 |q0: 0 |q1: 0 [2025-05-31 02:59:23.767806] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:23.777965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.782763] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] submit log success(service_type_=0, last_id=1748678380364653000, limited_id=1748678380364653000) [2025-05-31 02:59:23.782806] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] submit log with lock success(service_type_=0, last_id=1748678380364653000, limited_id=1748678380364653000) [2025-05-31 02:59:23.789523] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.790041] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38CA-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1748678380364653000, latest_scn={val:1748678363778615001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:59:23.790113] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38CA-0-0] [lt=70] handle submit callback(service_type_=0, success=true, used_ts=7352, log_ts={val:1748678363778615001, v:0}, limited_id=1748678380364653000) [2025-05-31 02:59:23.790173] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38CA-0-0] [lt=59] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:59:23.790185] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38CA-0-0] [lt=10] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:59:23.790213] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38CA-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678363789453000, start_id=1748678363789453000, end_id=1748678363789453001) [2025-05-31 02:59:23.798593] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:23.800735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.811619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.821938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.832282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.842716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.853409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.864382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.874989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.885954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.897021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.908694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.921096] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449C-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:23.921423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.931723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.935665] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=27] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:59:23.944167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.954397] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.965058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.968680] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=110] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:23.968898] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:23.976101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.986724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:23.999631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.000596] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-38121472, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:24.009958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.017553] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=91] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:24.022735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.033567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.046556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.056789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.059842] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:59:24.060160] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=99] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:59:24.067173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.077466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.078464] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-05-31 02:59:24.084576] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=29] current monitor number(seq_=-1) [2025-05-31 02:59:24.087779] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.088366] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=42] gc stale ls task succ [2025-05-31 02:59:24.098089] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.103864] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:24.107371] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=93] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:24.108453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=172][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.112228] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:24.118874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=237][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.129309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.133230] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1748678364133226, bytes: 7138293653, bw: 0.153807 MB/s, add_ts: 1005264, add_bytes: 162127 [2025-05-31 02:59:24.137350] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1748678364137347, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007303, add_bytes: 0 [2025-05-31 02:59:24.140983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.151213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.162662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.169054] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:24.169036] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=68, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:24.169179] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=123] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=4353, avg_group_log_size=334) [2025-05-31 02:59:24.169205] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4353) [2025-05-31 02:59:24.169248] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:24.174277] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4353) [2025-05-31 02:59:24.174488] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=339, avg_log_gen_to_submit_time=439, avg_log_submit_to_flush_time=3376, avg_log_submit_to_slide_time=5295) [2025-05-31 02:59:24.174617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.174614] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4353, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:24.179978] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=10] finish to resize log external storage handler(current_ts=1748678364179972, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:59:24.180059] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=78] snapshot_traffic_info once [2025-05-31 02:59:24.184822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.186776] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB75-0-0] [lt=11] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.606417655944824219e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=280, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.813808123964660808e+03) [2025-05-31 02:59:24.191297] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678363789453000, v:0}, new_max_version={val:1748678363789453000, v:0}, last_min_version={val:1748678363515365001, v:0}, last_max_version={val:1748678363515365001, v:0}, total_time_us=6869, delta_us=401839) [2025-05-31 02:59:24.191394] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678363789453000, v:0}, min_version={val:1748678363789453000, v:0}, max_version={val:1748678363789453000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:24.191419] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:24.195112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.201638] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:24.205606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.217110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.227351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.234615] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA8A-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:24.240028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.250298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.251091] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=125] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:24.260779] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.275261] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2077-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:24.275311] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2077-0-0] [lt=40] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=53) [2025-05-31 02:59:24.275618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.287179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.297416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.308033] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.318217] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=30] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:24.318431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.333360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.343565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.354005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.358969] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.359456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.359512] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.359593] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.364330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.369260] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:24.374675] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.375134] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=36] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:24.375161] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81063][TimerWK1_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:24.375296] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4E-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:24.375402] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4E-0-0] [lt=101] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:24.376913] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=2775, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:24.376950] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=325, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:24.376988] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3805) [2025-05-31 02:59:24.377011] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=31, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:24.377427] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC124B-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:24.377450] INFO [LIB] stat (utility.h:1211) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC124B-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5517, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:24.379385] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=42] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:24.385011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.394561] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [31227][T1_L0_G1][T1][YB42AC10871E-0006366549CC124C-0-0] [lt=133] init point thread id with(&point=0x55c0908ef280, idx_=2555, point=[thread id=31227, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=31227) [2025-05-31 02:59:24.395472] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:24.395527] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52482, [delay/us]:prepare:12, schedule:216, submit:49, rt:2060, total:2339) [2025-05-31 02:59:24.395544] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:24.395604] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52486, [delay/us]: prepare:12, schedule:216, submit:49, device:2060, total:2339) [2025-05-31 02:59:24.395618] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:24.395644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4037, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3076, total:3092) [2025-05-31 02:59:24.395668] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:24.395696] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4037, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3076, total:3092) [2025-05-31 02:59:24.395714] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:24.395728] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:24.395736] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:24.395745] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:24.395753] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:24.395762] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=148kB/s, net_out=98kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:24.395782] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399459][2nd=3055]}) [2025-05-31 02:59:24.396985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.405433] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:24.407199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.418229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.428483] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.439065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.444389] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=30] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:59:24.450387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.461069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.466054] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=30] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:24.467742] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=46] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:24.467868] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=44] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678362234341000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:24.467986] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:710) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=115] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1748678362234341000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1748678362234341000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}}) [2025-05-31 02:59:24.472056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.483773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.483884] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:24.490284] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [31227][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=52] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:24.494429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.500958] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:24.504707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.514435] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38CF-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525413696602, cache_obj->added_lc()=false, cache_obj->get_object_id()=53706, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:24.515983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.518376] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=60] ts response task statistics(alloc_count=486890, free_count=243444) [2025-05-31 02:59:24.518464] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:24.518483] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678364017616], gts:1748678364054858000, latest_srr:[mts=1748678364518413]}) [2025-05-31 02:59:24.527344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.537621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.548199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.558495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.568914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.571551] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:24.583011] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:24.583068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.590975] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.591028] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=70] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.591057] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.593284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.593607] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.593666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.593696] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.593739] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:24.603808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.604338] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=34] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:24.607382] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A75-0-0] [lt=32] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:24.609064] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=89] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-33927168, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:24.614247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.624458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.636047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.646815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.659480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.669621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.681968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.685820] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678364584942000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678364685770}, version_delta=100875) [2025-05-31 02:59:24.692204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.702407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.717465] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DC-0-0] [lt=95] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:24.719574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.729741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.741591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.751921] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.764108] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:24.764207] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=53] temporary file module use no blocks [2025-05-31 02:59:24.764222] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:24.764238] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:24.764252] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525412943664, flush_io_finished_ret_:0, flush_io_finished_round_:19932, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:24.764393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.772220] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=194] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:24.774614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.782837] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83091][T1_L0_G0][T1][YB42AC108714-000636671E8D5D5B-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678364782723000, start_id=1748678364782723000, end_id=1748678364782723001) [2025-05-31 02:59:24.783917] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:606000 disk:52K/12:606000 t1m2:52K/12:606000 t1m2g0:52K/12:606000 |q0: 0 |q1: 0 [2025-05-31 02:59:24.785170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:24.791199] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [31689][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:24.791254] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:24.794997] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC10871E-0006366542D8A73D-0-0] [lt=70] get number(ret=0, service_type_=0, range=1, base_id=1748678364791975000, start_id=1748678364791975000, end_id=1748678364791975001) [2025-05-31 02:59:24.801451] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.810009] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:24.812722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.823257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.833477] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.843746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.849485] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:24.849533] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=39] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=51970440, tenant_id=1) [2025-05-31 02:59:24.849557] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:24.853946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.864237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.875329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.884175] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=28] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19332075520, fast_delete_log_mode=false) [2025-05-31 02:59:24.885531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.896400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.907532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.918059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.923522] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449D-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:24.928263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.938734] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.949115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.960441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.967950] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=37] success to update min reserved snapshot(reserved_snapshot=1748676562234341000, duration=1800, snapshot_gc_ts=1748678362234341000) [2025-05-31 02:59:24.969497] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=35] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:24.970564] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:188) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=50] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1748676562234341000) [2025-05-31 02:59:24.970629] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=62] update reserved snapshot finished(cost_ts=2686, reserved_snapshot=1748676562234341000) [2025-05-31 02:59:24.971171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.972367] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:24.983520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:24.995567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.006270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.010781] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:25.016632] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.019363] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=55] get number(ret=0, service_type_=0, range=1, base_id=1748678365018189000, start_id=1748678365018189000, end_id=1748678365018189001) [2025-05-31 02:59:25.019414] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:25.019811] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678364518820], gts:1748678364556078000, latest_srr:[mts=1748678365019783]}) [2025-05-31 02:59:25.028070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.034498] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD42-0-0] [lt=63] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678365033294, keepalive_ts:-1}) [2025-05-31 02:59:25.034555] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28331][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD42-0-0] [lt=54] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678365033294, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:25.040169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.051025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.055934] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=42] ====== check clog disk timer task ====== [2025-05-31 02:59:25.055969] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=26] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:25.061798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.072063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=155][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.082492] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.085212] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:59:25.085324] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=14558208) [2025-05-31 02:59:25.086507] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=103] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678364791975000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=294521, in_cluster_service=true, cluster_version={val:1748678364785631000, v:0}, min_cluster_version={val:1748678364785631000, v:0}, max_cluster_version={val:1748678364785631000, v:0}, get_cluster_version_err=0, cluster_version_delta=300865, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195491, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=356031738, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:25.089729] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=24] gc stale ls task succ [2025-05-31 02:59:25.103628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.104711] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:25.109249] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:25.114596] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81742][T1_TimerWK1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=64] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:25.114657] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:25.114969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.125282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.136508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.136811] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678365136807, bytes: 7138451428, bw: 0.149929 MB/s, add_ts: 1003581, add_bytes: 157775 [2025-05-31 02:59:25.145399] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1748678365145394, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008047, add_bytes: 0 [2025-05-31 02:59:25.146785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.151990] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B02-0-0] [lt=25] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:25.157420] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B04-0-0] [lt=34] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:25.160120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.170328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.173657] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=487568, replace_num=15728) [2025-05-31 02:59:25.173692] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:25.180669] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:25.180790] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=13, avg_log_batch_cnt=1, total_group_log_size=5189, avg_group_log_size=399) [2025-05-31 02:59:25.180884] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=88] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5189) [2025-05-31 02:59:25.180956] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=68] [PALF STAT APPEND COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:25.181698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=167][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.186538] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5189) [2025-05-31 02:59:25.188990] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB84-0-0] [lt=51] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.606417655944824219e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=280, Throttle Trigger(MB)=168, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.813808123964660808e+03) [2025-05-31 02:59:25.189324] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61] get number(ret=0, service_type_=0, range=1, base_id=1748678365189132000, start_id=1748678365189132000, end_id=1748678365189132001) [2025-05-31 02:59:25.189638] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=103, avg_log_submit_to_flush_time=3325, avg_log_submit_to_slide_time=4429) [2025-05-31 02:59:25.189948] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5189, hit_cnt=13, read_cnt=13, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:25.192310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.193205] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678364791975002, v:0}, new_max_version={val:1748678364791975002, v:0}, last_min_version={val:1748678364785631000, v:0}, last_max_version={val:1748678364785631000, v:0}, total_time_us=6493, delta_us=401228) [2025-05-31 02:59:25.193241] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678364791975002, v:0}, min_version={val:1748678364791975002, v:0}, max_version={val:1748678364791975002, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:25.205231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.205609] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [31689][T1_L0_G1][T1][YB42AC1087A7-00063665517F5691-0-0] [lt=68] init point thread id with(&point=0x55c0908f6600, idx_=3017, point=[thread id=31689, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=31689) [2025-05-31 02:59:25.212983] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-40218624, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:25.217714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.228224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.240746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.252290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.253831] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83083][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA8B-0-0] [lt=13][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:25.263801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.274287] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.284674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.298619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.311483] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.320150] INFO dump_info (ob_timer_service.cpp:722) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump info [summary](this=0x2b08efdfeaf0, this={this:0x2b08efdfeaf0, tenant_id_:1, priority_task_queue_.size():57, running_task_set_.size():1, uncanceled_task_set_.size():0, worker_thread_pool_.get_queue_num():-3}) [2025-05-31 02:59:25.320189] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=38] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce658, timer:0x2b0902fffee0, task:0x2b08f41ab038, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr12TabletGCTaskE", scheduled_time:1748678365333133, pushed_time:0, delay:20000}) [2025-05-31 02:59:25.320357] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=166] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce498, timer:0x2b0902fffee0, task:0x2b08f41ab010, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr11TableGCTaskE", scheduled_time:1748678365333806, pushed_time:0, delay:20000}) [2025-05-31 02:59:25.320367] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=10] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ceff8, timer:0x2b0904973580, task:0x2b09049735f8, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService16ObCheckpointTaskE", scheduled_time:1748678365555307, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320376] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ced58, timer:0x2b08f12edf40, task:0x2b090496cbf0, task_type:"N9oceanbase7obmysql15ObEliminateTaskE", scheduled_time:1748678365585344, pushed_time:0, delay:500000}) [2025-05-31 02:59:25.320383] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cee38, timer:0x2b08f12edf40, task:0x2b090496cc40, task_type:"N9oceanbase7obmysql20ObConstructQueueTaskE", scheduled_time:1748678365604738, pushed_time:0, delay:500000}) [2025-05-31 02:59:25.320390] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce8f8, timer:0x2b08fe3d1ca0, task:0x2b090afc33c0, task_type:"N9oceanbase4palf10LogUpdaterE", scheduled_time:1748678365695783, pushed_time:0, delay:500000}) [2025-05-31 02:59:25.320398] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0a38, timer:0x2b08f12adde0, task:0x2b09061fb440, task_type:"N9oceanbase8observer28ObTenantLSMetaTableCheckTaskE", scheduled_time:1748678366031232, pushed_time:0, delay:0}) [2025-05-31 02:59:25.320405] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0bf8, timer:0x2b0902ffdee0, task:0x2b090499da98, task_type:"N9oceanbase3omt11ObTenantSrs27TenantSrsUpdatePeriodicTaskE", scheduled_time:1748678366131139, pushed_time:0, delay:0}) [2025-05-31 02:59:25.320412] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d06b8, timer:0x2b0902ffdee0, task:0x2b09049b2c38, task_type:"N9oceanbase5table21ObTableGroupCommitMgr19ObTableGroupOpsTaskE", scheduled_time:1748678366131329, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320419] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce1f8, timer:0x2b08f12c1f20, task:0x2b0904970b60, task_type:"N9oceanbase3sql27ObSqlPlanMonitorRecycleTaskE", scheduled_time:1748678366157556, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320426] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf298, timer:0x2b09061fbee8, task:0x2b09061fbf10, task_type:"N9oceanbase7storage10checkpoint17ObTabletGCService18ObTabletChangeTaskE", scheduled_time:1748678366176327, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320693] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=265] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cffb8, timer:0x2b090499cc70, task:0x2b090499cd10, task_type:"N9oceanbase8observer18ObTableLoadService21ObClientTaskPurgeTaskE", scheduled_time:1748678366189888, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320707] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=14] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfed8, timer:0x2b090499cc70, task:0x2b090499ccf8, task_type:"N9oceanbase8observer18ObTableLoadService25ObClientTaskAutoAbortTaskE", scheduled_time:1748678366305435, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320762] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=54] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfdf8, timer:0x2b090499cc70, task:0x2b090499cce0, task_type:"N9oceanbase8observer18ObTableLoadService13ObReleaseTaskE", scheduled_time:1748678366306107, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320771] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfb58, timer:0x2b090499cc70, task:0x2b090499cc98, task_type:"N9oceanbase8observer18ObTableLoadService17ObCheckTenantTaskE", scheduled_time:1748678366320103, pushed_time:0, delay:1000000}) [2025-05-31 02:59:25.320779] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ceab8, timer:0x2b08fe3cdef0, task:0x2b0904d8c518, task_type:"N9oceanbase7storage29ObTenantCheckpointSlogHandler21ObWriteCheckpointTaskE", scheduled_time:1748678366551692, pushed_time:0, delay:60000000}) [2025-05-31 02:59:25.320786] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf0d8, timer:0x2b09049735a8, task:0x2b0904973608, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService20ObTraversalFlushTaskE", scheduled_time:1748678366678223, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320793] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce3b8, timer:0x2b090c2ab330, task:0x2b090499d360, task_type:"N9oceanbase3sql16ObUDRRefreshTaskE", scheduled_time:1748678366771138, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320800] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce118, timer:0x2b08f12c1cd0, task:0x2b0904967b68, task_type:"N9oceanbase3sql26ObPlanCacheEliminationTaskE", scheduled_time:1748678366797302, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320808] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf1b8, timer:0x2b09049735d0, task:0x2b0904973618, task_type:"N9oceanbase7storage10checkpoint19ObCheckPointService24ObCheckClogDiskUsageTaskE", scheduled_time:1748678367056005, pushed_time:0, delay:2000000}) [2025-05-31 02:59:25.320815] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf998, timer:0x2b08f9bffde0, task:0x2b090499a038, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr10ReloadTaskE", scheduled_time:1748678367468026, pushed_time:0, delay:3000000}) [2025-05-31 02:59:25.320822] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d12f8, timer:0x2b0902ffdee0, task:0x2b09049ad9e0, task_type:"N9oceanbase10rootserver22ObMViewMaintenanceTaskE", scheduled_time:1748678367802683, pushed_time:0, delay:0}) [2025-05-31 02:59:25.320829] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf378, timer:0x2b09061fbf28, task:0x2b09061fbf50, task_type:"N9oceanbase7storage10checkpoint16ObEmptyShellTaskE", scheduled_time:1748678368685741, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320836] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cef18, timer:0x2b0902ffdee0, task:0x2b0904969980, task_type:"N9oceanbase3sql3dtl23ObDTLIntermResultGCTaskE", scheduled_time:1748678368922747, pushed_time:0, delay:10000000}) [2025-05-31 02:59:25.320843] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0258, timer:0x2b0902ffdee0, task:0x2b09049adcc0, task_type:"N9oceanbase10rootserver22ObMviewUpdateCacheTaskE", scheduled_time:1748678369198427, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320850] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d05d8, timer:0x2b0902ffdee0, task:0x2b09061ffd68, task_type:"N9oceanbase5share23ObStorageIOUsageRepoter23TenantIOUsageReportTaskE", scheduled_time:1748678369198814, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320857] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d1218, timer:0x2b0902ffdee0, task:0x2b09049adcd0, task_type:"N9oceanbase10rootserver16ObMViewMdsOpTaskE", scheduled_time:1748678369198974, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.320864] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0338, timer:0x2b0902ffdee0, task:0x2b0904967d00, task_type:"N9oceanbase5table21ObTableApiSessPoolMgr29ObTableApiSessEliminationTaskE", scheduled_time:1748678369199082, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.321093] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=228] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0418, timer:0x2b0902ffdee0, task:0x2b0904967d18, task_type:"N9oceanbase5table21ObTableApiSessPoolMgr30ObTableApiSessSysVarUpdateTaskE", scheduled_time:1748678369199096, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.321106] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=12] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce738, timer:0x2b08f1289cd0, task:0x2b09061fa5f0, task_type:"N9oceanbase12blocksstable21ObSharedMacroBlockMgr26ObBlockDefragmentationTaskE", scheduled_time:1748678369367683, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321113] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce038, timer:0x2b08f12adcd0, task:0x2b0906604038, task_type:"N9oceanbase7storage21ObTenantTabletStatMgr17TabletStatUpdaterE", scheduled_time:1748678369837936, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.321176] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=62] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ceb98, timer:0x2b08f12dbf40, task:0x2b0904967580, task_type:"N9oceanbase3sql24ObPsCacheEliminationTaskE", scheduled_time:1748678370114692, pushed_time:0, delay:5000000}) [2025-05-31 02:59:25.321185] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=9] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce578, timer:0x2b0902fffee0, task:0x2b08f41ab028, task_type:"N9oceanbase7storage18ObTenantMetaMemMgr17RefreshConfigTaskE", scheduled_time:1748678370270215, pushed_time:0, delay:10000000}) [2025-05-31 02:59:25.321192] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf618, timer:0x2b08f1259f40, task:0x2b0904991a58, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr18InfoPoolResizeTaskE", scheduled_time:1748678370332129, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321200] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfd18, timer:0x2b090499cc70, task:0x2b090499ccc8, task_type:"N9oceanbase8observer18ObTableLoadService8ObGCTaskE", scheduled_time:1748678370407273, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321207] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce9d8, timer:0x2b08f1289de0, task:0x2b09048044d0, task_type:"N9oceanbase10logservice17ReplayProcessStatE", scheduled_time:1748678370810537, pushed_time:0, delay:10000000}) [2025-05-31 02:59:25.321214] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfc38, timer:0x2b090499cc70, task:0x2b090499ccb0, task_type:"N9oceanbase8observer18ObTableLoadService15ObHeartBeatTaskE", scheduled_time:1748678373457346, pushed_time:0, delay:10000000}) [2025-05-31 02:59:25.321221] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cfa78, timer:0x2b08f9bffde0, task:0x2b090499a050, task_type:"N9oceanbase7storage21ObTenantFreezeInfoMgr24UpdateLSResvSnapshotTaskE", scheduled_time:1748678374970648, pushed_time:0, delay:10000000}) [2025-05-31 02:59:25.321228] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0878, timer:0x2b0902ffdee0, task:0x2b09049b35e0, task_type:"N9oceanbase5table20ObTableClientInfoMgr27ObTableClientInfoRetireTaskE", scheduled_time:1748678377270772, pushed_time:0, delay:60000000}) [2025-05-31 02:59:25.321235] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0b18, timer:0x2b0902ffdee0, task:0x2b08ffcce030, task_type:"N9oceanbase8observer20ObTableQueryASyncMgrE", scheduled_time:1748678377301223, pushed_time:0, delay:0}) [2025-05-31 02:59:25.321242] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf8b8, timer:0x2b08f12d5f40, task:0x2b0904991a78, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr15MediumCheckTaskE", scheduled_time:1748678381616776, pushed_time:0, delay:20000000}) [2025-05-31 02:59:25.321249] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf538, timer:0x2b08f1257f40, task:0x2b0904991a48, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr13SSTableGCTaskE", scheduled_time:1748678387841142, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321256] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d04f8, timer:0x2b0902ffdee0, task:0x2b0952fa16f0, task_type:"N9oceanbase8observer26ObTableLoadResourceManager21ObRefreshAndCheckTaskE", scheduled_time:1748678388912055, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321263] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0db8, timer:0x2b0902ffdee0, task:0x2b09049adbc8, task_type:"N9oceanbase10rootserver25ObMViewPushRefreshScnTaskE", scheduled_time:1748678388912324, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321270] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0798, timer:0x2b0902ffdee0, task:0x2b09049adbf8, task_type:"N9oceanbase10rootserver22ObReplicaSafeCheckTaskE", scheduled_time:1748678388915412, pushed_time:0, delay:0}) [2025-05-31 02:59:25.321277] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d1058, timer:0x2b0902ffdee0, task:0x2b09049adc90, task_type:"N9oceanbase10rootserver24ObCollectMvMergeInfoTaskE", scheduled_time:1748678388916218, pushed_time:0, delay:30000000}) [2025-05-31 02:59:25.321285] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0e98, timer:0x2b0902ffdee0, task:0x2b09049adbe0, task_type:"N9oceanbase10rootserver23ObMViewPushSnapshotTaskE", scheduled_time:1748678390328898, pushed_time:0, delay:60000000}) [2025-05-31 02:59:25.321292] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d1138, timer:0x2b0902ffdee0, task:0x2b09049adca8, task_type:"N9oceanbase10rootserver24ObMViewCleanSnapshotTaskE", scheduled_time:1748678390341772, pushed_time:0, delay:60000000}) [2025-05-31 02:59:25.321299] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf7d8, timer:0x2b08f12d5f40, task:0x2b0904991a38, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr14MediumLoopTaskE", scheduled_time:1748678401602489, pushed_time:0, delay:120000000}) [2025-05-31 02:59:25.321306] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062ce2d8, timer:0x2b08fe7fff40, task:0x2b0904973ac8, task_type:"N9oceanbase7storage25ObTenantCompactionMemPool17MemPoolShrinkTaskE", scheduled_time:1748678425582486, pushed_time:0, delay:120000000}) [2025-05-31 02:59:25.321313] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf458, timer:0x2b08f12cdf40, task:0x2b0904991a28, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr13MergeLoopTaskE", scheduled_time:1748678433474548, pushed_time:0, delay:120000000}) [2025-05-31 02:59:25.321372] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062cf6f8, timer:0x2b08f1259f40, task:0x2b0904991a68, task_type:"N9oceanbase10compaction30ObTenantTabletSchedulerTaskMgr24TabletUpdaterRefreshTaskE", scheduled_time:1748678488883689, pushed_time:0, delay:300000000}) [2025-05-31 02:59:25.321386] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=13] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0178, timer:0x2b0902ffdee0, task:0x2b09049a5be0, task_type:"N9oceanbase6common25ObOptStatMonitorCheckTaskE", scheduled_time:1748679092870970, pushed_time:0, delay:900000000}) [2025-05-31 02:59:25.321393] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=8] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0cd8, timer:0x2b08f12adef0, task:0x2b09061fb458, task_type:"N9oceanbase8observer32ObTenantTabletMetaTableCheckTaskE", scheduled_time:1748679992575396, pushed_time:0, delay:0}) [2025-05-31 02:59:25.321400] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0098, timer:0x2b0902ffdee0, task:0x2b09049a5bc8, task_type:"N9oceanbase6common28ObOptStatMonitorFlushAllTaskE", scheduled_time:1748744788731100, pushed_time:0, delay:86400000000}) [2025-05-31 02:59:25.321407] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0f78, timer:0x2b0902ffdee0, task:0x2b09049ad900, task_type:"N9oceanbase10rootserver21ObMLogMaintenanceTaskE", scheduled_time:1748746279653137, pushed_time:0, delay:0}) [2025-05-31 02:59:25.321414] INFO dump_info (ob_timer_service.cpp:725) [81745][T1_TimerSvr][T1][Y0-0000000000000000-0-0] [lt=7] dump queue token(this=0x2b08efdfeaf0, token={this:0x2b09062d0958, timer:0x2b0902ffdee0, task:0x2b09049adad8, task_type:"N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE", scheduled_time:1748746279666935, pushed_time:0, delay:0}) [2025-05-31 02:59:25.322586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.337096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.347434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.358272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=232][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.358389] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.359603] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.359674] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.361804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=84] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.371072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.373939] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:25.382114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.386069] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [30620][T1_L0_G1][T1][YB42AC1087A7-00063665517F5693-0-0] [lt=54] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5735) [2025-05-31 02:59:25.386332] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=106] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2560, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:25.386424] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=82] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=683, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:25.386482] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:25.386500] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4608, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:25.386519] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=40, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:25.387103] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] no need to update min/max version(new_version={val:1748678364791975002, v:0}, min_version={val:1748678364791975002, v:0}, max_version={val:1748678364791975002, v:0}) [2025-05-31 02:59:25.387139] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:25.393054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.401307] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:25.401372] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4726, iops:13, norm_iops:3, bw:61082, [delay/us]:prepare:12, schedule:502, submit:76, rt:1985, total:2576) [2025-05-31 02:59:25.401437] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:25.401600] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=145] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4726, avg_iops:13, avg_bw:61073, [delay/us]: prepare:12, schedule:502, submit:76, device:1985, total:2576) [2025-05-31 02:59:25.401618] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:25.401660] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4071, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:968, total:982) [2025-05-31 02:59:25.401687] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:25.401720] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4071, [delay/us]: prepare:0, schedule:0, submit:600000000, device:968, total:982) [2025-05-31 02:59:25.401738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:25.401752] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:25.401761] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:25.401769] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:25.401777] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:25.401786] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=179kB/s, net_out=119kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:25.401807] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399460][2nd=2632]}) [2025-05-31 02:59:25.404234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=153][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.416453] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=192] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-38121472, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:25.416997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.419816] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2025-05-31 02:59:25.427764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.441295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=212][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.452091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.462443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.472667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.487552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=176][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.497848] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.508733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=230][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.514697] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83083][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA8B-0-0] [lt=120][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525414696864, cache_obj->added_lc()=false, cache_obj->get_object_id()=53707, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.519009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.523499] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38D1-0-0] [lt=36][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525414705668, cache_obj->added_lc()=false, cache_obj->get_object_id()=53708, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.523756] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=677, get_gts_with_stc_cnt=50, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=48, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:25.530471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.540913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=198][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.553981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.555610] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=26] ====== checkpoint timer task ====== [2025-05-31 02:59:25.556587] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=26] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:25.556744] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=156] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:25.557203] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=57] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:25.557301] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=91] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:25.557363] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=53] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:25.557379] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678363778615001, v:0}) [2025-05-31 02:59:25.557388] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:25.557396] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:25.557411] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:25.557433] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:25.557446] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:25.557467] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:25.557490] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:25.557501] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:25.564783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.574192] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:25.575276] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.586085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.586309] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] sql audit evict mem start(request_manager_->get_tenant_id()=1, evict_low_mem_level=0, evict_high_mem_level=16106127, size_used=0, mem_used=16637952) [2025-05-31 02:59:25.587810] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=2, elapse_time=1501, size_used=0, mem_used=0) [2025-05-31 02:59:25.591228] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.591280] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.591309] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.591336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.593480] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.593546] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.593577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:25.596415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.604899] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=93] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:25.605645] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:25.605738] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.605780] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.605804] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.607140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.615724] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] start to print blacklist info [2025-05-31 02:59:25.616383] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=48] ls blacklist refresh finish(cost_time=35818) [2025-05-31 02:59:25.617446] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.620435] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:25.627807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=136][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.639130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=190][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.652753] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.664821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.675340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.688194] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678365201538002, v:0}, base_version={val:1748678365189132000, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678365486568000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678365688173, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678365201538002, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678365673630, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678365201538002, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678365684915, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678360835507000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678365657633, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678360673747000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678365654645, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678360799530000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678365623857, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:59:25.690206] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC108714-000636671E6D5146-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525414872377, cache_obj->added_lc()=false, cache_obj->get_object_id()=53709, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:25.690427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=378][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.700799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.706126] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81744][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:25.711411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.721964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.732755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.743054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=145][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.753346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.764092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.767017] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:25.767058] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-31 02:59:25.767070] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:25.767086] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:25.767102] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525413946465, flush_io_finished_ret_:0, flush_io_finished_round_:19933, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:25.774618] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=30][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:25.775133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.775316] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:25.785360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.785429] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:25.786206] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [29190][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=8] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:25.795995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.803443] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1F17-0-0] [lt=37] get number(ret=0, service_type_=0, range=1, base_id=1748678365800980000, start_id=1748678365800980001, end_id=1748678365800980002) [2025-05-31 02:59:25.806429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.816751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.821050] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x55c090a74fc0, last_check_time:1748678365821045, last_idle_time:78861268407, last_thread_count:4}, *pool={name:TimerWK, this:0x55c090a74fc0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:78861268407, tenant_id:500}, qsize=-3, current_idle_time=78861268407) [2025-05-31 02:59:25.821110] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=59] [thread pool stat](pool_stat={pool:0x2b08f13edcf0, last_check_time:1748678365821108, last_idle_time:42509206228, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13edcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:42509206228, tenant_id:500}, qsize=-1, current_idle_time=42509206228) [2025-05-31 02:59:25.821126] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] [thread pool stat](pool_stat={pool:0x2b08f13eeaf0, last_check_time:1748678365821125, last_idle_time:19981319367, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13eeaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:19981319367, tenant_id:500}, qsize=-1, current_idle_time=19981319367) [2025-05-31 02:59:25.821137] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2b08efdfee70, last_check_time:1748678365821136, last_idle_time:77370996717, last_thread_count:4}, *pool={name:TimerWK, this:0x2b08efdfee70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:77370996717, tenant_id:1}, qsize=-4, current_idle_time=77370996717) [2025-05-31 02:59:25.821147] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2b08f13ef1f0, last_check_time:1748678365821147, last_idle_time:2808737127, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2808737127, tenant_id:500}, qsize=-1, current_idle_time=2808737127) [2025-05-31 02:59:25.821157] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [81110][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2b08f13ef8f0, last_check_time:1748678365821156, last_idle_time:2808305468, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2b08f13ef8f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:2808305468, tenant_id:500}, qsize=-1, current_idle_time=2808305468) [2025-05-31 02:59:25.823340] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:25.827123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.837453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.848527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.859056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.869445] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=118][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.880994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.891188] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=70] no need to update min/max version(new_version={val:1748678365201538002, v:0}, min_version={val:1748678365201538002, v:0}, max_version={val:1748678365201538002, v:0}) [2025-05-31 02:59:25.891593] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=40] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:56K/13:125507 disk:56K/13:125507 t1m2:56K/13:125507 t1m2g0:56K/13:125507 |q0: 0 |q1: 0 [2025-05-31 02:59:25.891636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.901886] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.912294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.922586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.926137] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:25.932935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.943259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.953653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.964023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.965267] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=28] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:59:25.972257] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=15] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:25.974363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.975558] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:25.984744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:25.996646] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.006870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.017190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.024007] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:26.024961] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:26.025277] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:26.027796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.040363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.051103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.062110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.074057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.085030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.088269] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=119] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:26.089879] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=28] gc stale ls task succ [2025-05-31 02:59:26.092534] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:26.092579] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=43] try gc retain ctx [2025-05-31 02:59:26.092589] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-05-31 02:59:26.092643] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1748678365995898000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678355989106000, v:0}, keep_alive_scn_:{val:1748678361008386000, v:0}, update_ts_:1748678366092016}) [2025-05-31 02:59:26.093689] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:26.097766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:26.103068] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [32115][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:26.103214] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=116] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:26.105824] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=84] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:26.109434] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:26.113480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.115079] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:26.123731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.133924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.139038] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=89] PNIO [ratelimit] time: 1748678366139034, bytes: 7138626778, bw: 0.166855 MB/s, add_ts: 1002227, add_bytes: 175350 [2025-05-31 02:59:26.144165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.155208] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678366155161, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009767, add_bytes: 0 [2025-05-31 02:59:26.155429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.160178] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B41-0-0] [lt=8] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:26.167279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.178283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.178496] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=125] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:26.178608] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:26.178634] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81743][T1_TimerWK2_][T1][Y0-0000000000000000-0-0] [lt=22] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=956) [2025-05-31 02:59:26.178690] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=76] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:26.189327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.195808] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=113] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=81, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:26.195852] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=3747, avg_group_log_size=312) [2025-05-31 02:59:26.195936] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=76] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3747) [2025-05-31 02:59:26.195952] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:26.196692] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB8A-0-0] [lt=91] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.726416969299316406e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.587160814026798562e+03) [2025-05-31 02:59:26.200021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.204719] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3747) [2025-05-31 02:59:26.205229] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=241, avg_log_gen_to_submit_time=342, avg_log_submit_to_flush_time=4223, avg_log_submit_to_slide_time=4878) [2025-05-31 02:59:26.205564] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3747, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:26.210352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.212338] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678365995898000, v:0}, new_max_version={val:1748678365995898000, v:0}, last_min_version={val:1748678365687382000, v:0}, last_max_version={val:1748678365687382000, v:0}, total_time_us=17286, delta_us=216436) [2025-05-31 02:59:26.212382] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678365995898000, v:0}, min_version={val:1748678365995898000, v:0}, max_version={val:1748678365995898000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:26.220617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.226071] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:26.231318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.241601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.254227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.265317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.275679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.287512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.289069] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2078-0-0] [lt=76] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:26.289106] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2078-0-0] [lt=27] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-05-31 02:59:26.298476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=609][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.301297] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA8C-0-0] [lt=53][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:26.309135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.320024] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.330741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.338690] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC108714-000636671EBE4281-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1748678366329573000, start_id=1748678366329573000, end_id=1748678366329573001) [2025-05-31 02:59:26.351917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.362405] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.363337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.363417] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.363449] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.363609] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.374139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.377535] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=66] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:26.377564] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=28] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:26.377898] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4F-0-0] [lt=18] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:26.377936] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F4F-0-0] [lt=32] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:26.378379] INFO pn_ratelimit (group.c:653) [81063][TimerWK1_IngressService][T0][Y0-0000000000000000-0-0] [lt=25] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:26.383734] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=101] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:26.384780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.395277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.401681] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [28331][T1_L0_G1][T1][YB42AC10871E-0006366549CC1264-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3201) [2025-05-31 03:59:26.408932] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [32115][T1_L0_G1][T1][YB42AC1087A7-00063665517F569E-0-0] [lt=45] init point thread id with(&point=0x55c0908fd080, idx_=3443, point=[thread id=32115, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=32115) [2025-05-31 02:59:26.409822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.410052] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:26.410175] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:10, norm_iops:2, bw:40604, [delay/us]:prepare:17, schedule:1047, submit:443, rt:2758, total:4267) [2025-05-31 02:59:26.410215] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:26.410309] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=72] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:10, avg_bw:40606, [delay/us]: prepare:17, schedule:1047, submit:443, device:2758, total:4267) [2025-05-31 02:59:26.410331] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:26.410369] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4060, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:441, total:465) [2025-05-31 02:59:26.410408] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:26.410673] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=257] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4059, [delay/us]: prepare:0, schedule:0, submit:600000000, device:441, total:465) [2025-05-31 02:59:26.410701] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:26.410731] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:26.410751] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:26.410771] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:26.410787] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:26.410800] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SERVER] net_in=165kB/s, net_out=140kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:26.410828] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] successfully alloc block(macro_id={[ver=1,mode=0,seq=399461][2nd=2504]}) [2025-05-31 02:59:26.416619] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5587, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:26.416724] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=101][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137422888}], scn_array=[cnt:1, {val:1748678366197065001, v:0}], curr_size=122, accum_size=711, time_cost=17659) [2025-05-31 02:59:26.416778] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=396, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:26.417401] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:26.417423] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7786, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:26.417443] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=100, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:26.423148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.427481] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:26.428957] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:26.433404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.443990] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.454245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.467155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.477497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.486016] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=57] new mysql sessid created(conn.sessid_=3221740091, support_ssl=false) [2025-05-31 02:59:26.486168] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221740091, sess.client_addr_="172.16.135.166:43898") [2025-05-31 02:59:26.486191] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x2b08ff896450, session_id:3221740091, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:141}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.487169] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81570][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=100] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:59:26.487188] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [81570][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] mysql login queue(mysql_queue->queue_.size()=1) [2025-05-31 02:59:26.487257] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCE-0-0] [lt=27] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221740091, conn.client_addr_port_=43898, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221740091, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221740091, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-0006366544F9FFCE-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82989, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418709035451540, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:82, fixup_index:-1, fixup_ash_buffer:{this:0x2b0968765c58, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:59:26.488631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.489413] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCE-0-0] [lt=119][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:59:26.489550] INFO [SERVER] process (obmp_connect.cpp:504) [82989][T1_MysqlQueueTh][T1][Y0-0006366544F9FFCE-0-0] [lt=20] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221740091, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:59:26.490303] INFO [SERVER] process (obmp_init_db.cpp:44) [83080][T1_L0_G0][T1][YB42AC1087A6-00063665502A138E-0-0] [lt=45] init db(db_name=oceanbase) [2025-05-31 02:59:26.499003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.510407] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=69] new mysql sessid created(conn.sessid_=3221739965, support_ssl=false) [2025-05-31 02:59:26.510480] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=39] sm conn init succ(conn.sessid_=3221739965, sess.client_addr_="172.16.135.166:43902") [2025-05-31 02:59:26.510484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.510498] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x2b08ff8971d0, session_id:3221739965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.511229] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81568][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:59:26.511599] INFO [SERVER] check_common_property (obmp_connect.cpp:2041) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE6-0-0] [lt=31] construct session id(conn.client_sessid_=1531445305, conn.sessid_=3221739965, conn.client_addr_port_=43902, conn.client_create_time_=1748674643592435, conn.proxy_sessid_=12398558917079269387, ObLocalDiagnosticInfo::get()={tenant_id:1, group_id:0, session_id:3221739965, curr_wait:event_no:0, p1:0, p2:0, p3:0, wait_begin_time:0, wait_end_time:0, wait_time:0, timeout_ms:0, level:0, parent:0, event_id:10000, event_name:"", param1:"", param2:"", param3:"", ref_cnt:2, need_aggregate:true, get_uref():1073741824, get_href():2, ash_stat:{ObActiveSessionStatItem:{tenant_id:1, session_id:3221739965, event id:10000, event:"", wait_time:0, time_model:0, trace_id:Y0-000636654349FFE6-0-0, plan_line_id:-1, sql_id:"", top_level_sql_id:"", plsql_entry_subprogram_name:"", plsql_subprogram_name:"", session_type:0, is_wr_sample:false, delta_time:0, delta_cpu_time:0, delta_db_time:0, program:"T1_SQL_CMD", module:"", action:"", client_id:"", group_id:0, tid:82987, plan_hash:0, tx_id:0, stmt_type:0, tablet_id:0, block_sessid:0}, last_touch_ts:1418709099843276, wait_event_begin_ts:0, total_idle_wait_time:0, total_non_idle_wait_time:0, prev_idle_wait_time:0, prev_non_idle_wait_time:0, total_cpu_time:0, is_active_session:true, inner_sql_wait_type_id:0, pcode:0, tm_idle_time:577, fixup_index:-1, fixup_ash_buffer:{this:0x2b09aa70ec38, block_ptr_.control_ptr:null, block_ptr_.data_ptr:null}, retry_wait_event_no:0, retry_plan_line_id:-1}}) [2025-05-31 02:59:26.517327] WDIAG [RPC.OBMYSQL] serialize (ompk_ok.cpp:261) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE6-0-0] [lt=148][errcode=-4019] size is overflow(length=13312, pos=4, need_size=23515, ret=-4019) [2025-05-31 02:59:26.517586] INFO [SERVER] process (obmp_connect.cpp:504) [82987][T1_MysqlQueueTh][T1][Y0-000636654349FFE6-0-0] [lt=28] MySQL LOGIN(direct_client_ip="172.16.135.166", client_ip=172.16.135.166, tenant_name=sys, tenant_id=1, user_name=proxyro, host_name=%, sessid=3221739965, proxy_sessid=12398558917079269387, client_sessid=1531445305, from_proxy=true, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=150972045, proxy_capability=12172103, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:59:26.522642] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=10] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:26.522958] INFO [SERVER] process (obmp_init_db.cpp:44) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654CCA1C84-0-0] [lt=51] init db(db_name=oceanbase) [2025-05-31 02:59:26.524961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.530326] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=87] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678366530287], gts:1748678366524833000, latest_srr:[mts=1748678366530287]}) [2025-05-31 02:59:26.538595] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:26.541460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.552200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.555131] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=74] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:26.558552] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=47] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:26.562433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.562561] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=47] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:26.562586] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=24] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:26.564947] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:26.565064] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=41] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:26.565096] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=29] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:26.565164] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=51] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:26.565994] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=824] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:26.566037] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=33] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:26.566049] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:26.566072] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:26.566093] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:26.568143] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:26.568173] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=26] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:26.568190] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=14] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:26.568251] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=12] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:26.568271] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=19] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:26.568281] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:26.568294] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:26.568311] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE0-0-0] [lt=16] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:26.572769] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.573234] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff8971d0, session_id:3221739965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678366524833, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:822, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.573342] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=94] kill and revert session(conn.sessid_=3221739965, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:59:26.573361] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sock ref clean, do destroy(*s={this:0x2b08ff8971d0, session_id:3221739965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:143}, err:5, last_decode_time:1748678366524833, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:822, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.573460] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=68] unlock session of tenant(conn.sessid_=3221739965, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:59:26.573479] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81568][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221739965, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654CCA1C86-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:13488905, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:13488906, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:282, pkt_seq:0, req_id:13488907, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:278, rec_:282, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:13488907, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:13488907, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:7, seq_:9}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:13488907, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:10}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:773, com_seq:1}, obp20_header_:{payload_len:745, pkt_seq:1, req_id:13488907, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:16}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:13488907, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:17}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:13488907, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:18}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:59:26.573829] INFO [SERVER] run (obmp_disconnect.cpp:78) [83092][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=70] free session successfully(sessid=3221739965) [2025-05-31 02:59:26.573949] INFO [SERVER] run (obmp_disconnect.cpp:85) [83092][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=117] mark session id unused(sessid=3221739965) [2025-05-31 02:59:26.574181] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff896450, session_id:3221740091, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678366491280, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.574225] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221740091, proxy_sessid=12398558917079269387, ret=0) [2025-05-31 02:59:26.574236] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sock ref clean, do destroy(*s={this:0x2b08ff896450, session_id:3221740091, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678366491280, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:702, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:26.574293] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] unlock session of tenant(conn.sessid_=3221740091, proxy_sessid=12398558917079269387, conn.tenant_id_=1) [2025-05-31 02:59:26.574307] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81570][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221740091, proxy_sessid=12398558917079269387, tenant_id=1, from_proxy=true, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-00063665502A1390-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:8, cur_pkt_pos_:10, pkt_rec[0]:{send:obp_mysql_header_{len_:23519, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp20_header_:{payload_len:14, pkt_seq:0, req_id:10918705, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:651, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{send:obp20_header_:{payload_len:43, pkt_seq:0, req_id:10918706, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:626, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[3]:{receive:obp20_header_:{payload_len:162, pkt_seq:0, req_id:10918707, flag_.flags:26, ext_flags:0}, obp_mysql_header_{len_:158, rec_:162, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[4]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:10918707, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:10918707, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:8}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:10918707, flag_.flags:8, ext_flags:0}, obp_mysql_header_{len_:9, seq_:9}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_compess_header_{com_len:547, com_seq:1}, obp20_header_:{payload_len:519, pkt_seq:1, req_id:10918707, flag_.flags:8, ext_flags:0}, obp_mysql_header_{pkt_num_:3, seq_:12}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:10918707, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:35, seq_:13}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[9]:{send:obp_compess_header_{com_len:75, com_seq:2}, obp20_header_:{payload_len:47, pkt_seq:2, req_id:10918707, flag_.flags:11, ext_flags:0}, obp_mysql_header_{len_:12, seq_:14}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:59:26.574491] INFO [SERVER] run (obmp_disconnect.cpp:78) [83082][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=130] free session successfully(sessid=3221740091) [2025-05-31 02:59:26.574514] INFO [SERVER] run (obmp_disconnect.cpp:85) [83082][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=22] mark session id unused(sessid=3221740091) [2025-05-31 02:59:26.579230] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83103][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38D3-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525415761398, cache_obj->added_lc()=false, cache_obj->get_object_id()=53710, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:26.583364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.589670] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:26.589802] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=103] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:26.593966] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.594023] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.594052] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.594421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.594783] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.594901] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.594932] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.595223] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:26.598949] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] no need to update min/max version(new_version={val:1748678366197065000, v:0}, min_version={val:1748678366197065000, v:0}, max_version={val:1748678366197065000, v:0}) [2025-05-31 02:59:26.606020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.607205] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=124] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:26.613529] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=454][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137423599}], scn_array=[cnt:1, {val:1748678366524833000, v:0}], curr_size=122, accum_size=1422, time_cost=11750) [2025-05-31 02:59:26.617416] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38D5-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525415799585, cache_obj->added_lc()=false, cache_obj->get_object_id()=53711, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:26.617836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.619055] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38D7-0-0] [lt=46][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:26.620728] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38D9-0-0] [lt=67][errcode=-4201] hashtable not init [2025-05-31 02:59:26.621756] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1748678366621603000, start_id=1748678366621603000, end_id=1748678366621603001) [2025-05-31 02:59:26.628414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.628447] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:26.628712] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:525) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=60] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 22887424 | cache_store_size= 20807680 | cache_map_size= 2079744 | kv_cnt= 1355 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tablet_table_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 52 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 426 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 295 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 473 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= bf_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= fuse_row_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 500 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 10402816 | cache_store_size= 8323072 | cache_map_size= 2079744 | kv_cnt= 968 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 3 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= log_kv_cache | cache_size= 8322048 | cache_store_size= 6242304 | cache_map_size= 2079744 | kv_cnt= 79 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 179 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1358 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_ds_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 27 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_system_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 1 | hold_size= 0 [2025-05-31 02:59:26.628741] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] [KVCACHE-HAZARD] hazard version status info: current_version: 1732626 | min_version= 1732626 | total_nodes_count: 0 | [KVCACHE-HAZARD] i= 2 | acquire_version=18446744073709551615 | waiting_nodes_count= 14 | last_retire_version= 1730974 | [KVCACHE-HAZARD] i= 658 | acquire_version=18446744073709551615 | waiting_nodes_count= 2 | last_retire_version= 1730974 | [2025-05-31 02:59:26.633721] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137423721}], scn_array=[cnt:1, {val:1748678366621603000, v:0}], curr_size=566, accum_size=1988, time_cost=11309) [2025-05-31 02:59:26.640088] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.651738] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC10871E-000636654E189224-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1748678366651622000, start_id=1748678366651622000, end_id=1748678366651622001) [2025-05-31 02:59:26.651876] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.662256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.672727] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=269][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.682311] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81743][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=317] ====== traversal_flush timer task ====== [2025-05-31 02:59:26.682738] INFO [MDS] flush (mds_table_mgr.cpp:184) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF5-0-0] [lt=38] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678366399266000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:26.682971] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF5-0-0] [lt=224] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:26.683278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.693657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.704187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=155][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.714750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.724675] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DD-0-0] [lt=45] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:26.724992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.735666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.745993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.758104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.769022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.771400] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:26.774199] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F47-0-0] [lt=25] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=38) [2025-05-31 02:59:26.774529] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678366773840000, start_id=1748678366773840000, end_id=1748678366773840001) [2025-05-31 02:59:26.780041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.790291] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:26.790360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.797967] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=19] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:26.798009] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=35] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:26.798024] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:26.798035] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:26.800780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.800976] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=314, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53694, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:26.801805] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81740][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=823] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:26.813217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.823696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.829222] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=130] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:26.833951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.844202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.854666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.864866] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.867597] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:26.867671] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=62] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:59:26.867697] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=16] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:26.876511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.886932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.897186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.904462] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:1042996 disk:52K/12:1042996 t1m2:52K/12:1042996 t1m2g0:52K/12:1042996 |q0: 0 |q1: 0 [2025-05-31 02:59:26.909552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.917285] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83074][T1_L0_G0][T1][YB42AC10870A-000636671698E3F4-0-0] [lt=63][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525416099455, cache_obj->added_lc()=false, cache_obj->get_object_id()=53712, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:26.919778] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.928477] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A449F-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:26.930128] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.940361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.950578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.960883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.971819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.972510] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=31] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:26.982421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:26.990528] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:26.991417] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=52] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:26.992810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.003386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.013578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.024008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.029786] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:27.032993] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=126] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:27.034402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.039810] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81064][TimerWK2_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=46] runTimer to refresh locality_info(ret=0) [2025-05-31 02:59:27.042716] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=12] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:59:27.042804] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=73] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:59:27.042819] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:59:27.042923] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:59:27.042959] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=34] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:59:27.042986] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:59:27.043002] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=17] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:59:27.043015] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=6] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:59:27.044561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.054913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.057048] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=10] ====== check clog disk timer task ====== [2025-05-31 02:59:27.057164] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=105] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:27.065192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.075341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.085662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.085957] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=30] current monitor number(seq_=-1) [2025-05-31 02:59:27.089958] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=28] gc stale ls task succ [2025-05-31 02:59:27.090455] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-05-31 02:59:27.097663] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.110690] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.110738] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:27.111305] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [28331][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=445] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:27.112394] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:27.112535] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:27.115247] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:27.121300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=254][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.132150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.141713] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1748678367141709, bytes: 7138838010, bw: 0.200909 MB/s, add_ts: 1002675, add_bytes: 211232 [2025-05-31 02:59:27.142946] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [32115][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA5-0-0] [lt=94] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678367178247, keepalive_ts:-1}) [2025-05-31 02:59:27.143047] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [32115][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA5-0-0] [lt=99] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678367178247, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:27.143186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.155558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.160355] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1748678367160350, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005189, add_bytes: 0 [2025-05-31 02:59:27.165704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.169835] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81433][pnio1][T0][YB42AC10870A-000636671AB947C8-0-0] [lt=96][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:512135041, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678367203532, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1390327, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678367180571, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678367169828, enqueue_timestamp_:0, request_arrival_time_:1748678367169828, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:27.170082] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81433][pnio1][T0][YB42AC10870A-000636671AB947C8-0-0] [lt=137][errcode=-5150] deliver rpc request fail(&req=0x2b0984204098, ret=-5150) [2025-05-31 02:59:27.176079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.186589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.191827] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:27.197008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.207569] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=100, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:27.207612] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4545, avg_group_log_size=413) [2025-05-31 02:59:27.207643] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4545) [2025-05-31 02:59:27.207676] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:27.207737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.210666] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4545) [2025-05-31 02:59:27.212425] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32115][T1_L0_G1][T1][YB42AC10871E-0006366549CC126D-0-0] [lt=21] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=2222, avg_log_gen_to_submit_time=2346, avg_log_submit_to_flush_time=9094, avg_log_submit_to_slide_time=9818) [2025-05-31 02:59:27.213210] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4545, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:27.217945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.229300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.230372] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:27.239513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.249798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.260084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.271232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.281545] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.291972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.302642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.306713] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFB95-0-0] [lt=77] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.726416969299316406e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.587160814026798562e+03) [2025-05-31 02:59:27.315825] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678367104901000, v:0}, new_max_version={val:1748678367104901000, v:0}, last_min_version={val:1748678366902150000, v:0}, last_max_version={val:1748678366902150000, v:0}, total_time_us=11307, delta_us=210920) [2025-05-31 02:59:27.315972] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=142] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678367104901000, v:0}, min_version={val:1748678367104901000, v:0}, max_version={val:1748678367104901000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:27.316020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.328278] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83105][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA8D-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:27.328738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.339504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.354502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.359106] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.359148] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.359173] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.359419] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.364889] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.376338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.386864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.392873] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=503296, replace_num=15728) [2025-05-31 02:59:27.393005] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=127] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:27.397069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.407326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.409851] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4545) [2025-05-31 02:59:27.415809] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:27.415932] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52943, [delay/us]:prepare:26, schedule:358, submit:251, rt:5179, total:5815) [2025-05-31 02:59:27.415955] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:27.416012] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52945, [delay/us]: prepare:26, schedule:358, submit:251, device:5179, total:5815) [2025-05-31 02:59:27.416025] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:27.416050] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4072, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:903, total:936) [2025-05-31 02:59:27.416075] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:27.416103] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4073, [delay/us]: prepare:0, schedule:0, submit:600000000, device:903, total:936) [2025-05-31 02:59:27.416121] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:27.416135] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:27.416143] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:27.416152] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:27.416160] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:27.416169] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=202kB/s, net_out=113kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:27.416189] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399462][2nd=2786]}) [2025-05-31 02:59:27.422170] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83085][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33DE-0-0] [lt=44][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525416604339, cache_obj->added_lc()=false, cache_obj->get_object_id()=53713, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:27.423832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.429080] INFO [STORAGE] get_macro_block_list (ob_sn_tmp_file_manager.cpp:187) [81064][TimerWK2_BlkMgr][T1][Y0-0000000000000000-0-0] [lt=95] get tmp file macro block list(ret=0, ret="OB_SUCCESS", macro_id_list.count()=0) [2025-05-31 02:59:27.432309] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:27.434147] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2025-05-31 02:59:27.434267] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=32] block manager free block(macro_id={[ver=1,mode=0,seq=399434][2nd=2426]}, io_fd={first_id:1152921504607246410, second_id:2426, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434294] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] block manager free block(macro_id={[ver=1,mode=0,seq=399433][2nd=2144]}, io_fd={first_id:1152921504607246409, second_id:2144, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434305] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] block manager free block(macro_id={[ver=1,mode=0,seq=399435][2nd=2862]}, io_fd={first_id:1152921504607246411, second_id:2862, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434314] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399436][2nd=2157]}, io_fd={first_id:1152921504607246412, second_id:2157, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434322] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399437][2nd=2439]}, io_fd={first_id:1152921504607246413, second_id:2439, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434330] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399438][2nd=2721]}, io_fd={first_id:1152921504607246414, second_id:2721, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434338] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399439][2nd=3003]}, io_fd={first_id:1152921504607246415, second_id:3003, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434346] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399440][2nd=2298]}, io_fd={first_id:1152921504607246416, second_id:2298, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434354] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399441][2nd=2580]}, io_fd={first_id:1152921504607246417, second_id:2580, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434362] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399442][2nd=2452]}, io_fd={first_id:1152921504607246418, second_id:2452, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434369] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399443][2nd=3016]}, io_fd={first_id:1152921504607246419, second_id:3016, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434377] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399444][2nd=1465]}, io_fd={first_id:1152921504607246420, second_id:1465, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434385] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399445][2nd=2311]}, io_fd={first_id:1152921504607246421, second_id:2311, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434393] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399446][2nd=2593]}, io_fd={first_id:1152921504607246422, second_id:2593, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434400] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399447][2nd=2875]}, io_fd={first_id:1152921504607246423, second_id:2875, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434408] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399448][2nd=3029]}, io_fd={first_id:1152921504607246424, second_id:3029, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434416] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399451][2nd=2606]}, io_fd={first_id:1152921504607246427, second_id:2606, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434424] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399450][2nd=2888]}, io_fd={first_id:1152921504607246426, second_id:2888, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434432] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399452][2nd=2337]}, io_fd={first_id:1152921504607246428, second_id:2337, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434440] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399453][2nd=2619]}, io_fd={first_id:1152921504607246429, second_id:2619, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434448] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399454][2nd=2901]}, io_fd={first_id:1152921504607246430, second_id:2901, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434455] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399455][2nd=2760]}, io_fd={first_id:1152921504607246431, second_id:2760, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434463] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399456][2nd=2914]}, io_fd={first_id:1152921504607246432, second_id:2914, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434471] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399457][2nd=2491]}, io_fd={first_id:1152921504607246433, second_id:2491, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434479] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399458][2nd=2773]}, io_fd={first_id:1152921504607246434, second_id:2773, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434486] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399459][2nd=3055]}, io_fd={first_id:1152921504607246435, second_id:3055, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434494] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399460][2nd=2632]}, io_fd={first_id:1152921504607246436, second_id:2632, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434501] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399461][2nd=2504]}, io_fd={first_id:1152921504607246437, second_id:2504, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434509] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] block manager free block(macro_id={[ver=1,mode=0,seq=399462][2nd=2786]}, io_fd={first_id:1152921504607246438, second_id:2786, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434518] INFO [STORAGE.BLKMGR] sweep_one_block (ob_block_manager.cpp:939) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] block manager free block(macro_id={[ver=1,mode=0,seq=399449][2nd=1840]}, io_fd={first_id:1152921504607246425, second_id:1840, third_id:-1, fd_id:-1, slot_version:-1, device_handle:null}) [2025-05-31 02:59:27.434616] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1023) [81064][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] finish once mark and sweep(ret=0, alloc_num=42488, mark_info.count()=30, marker_status={total_block_count:3072, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:31, shared_data_block_count:12, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:0, pending_free_count:30, free_count:3013, shared_meta_block_count:14, mark_cost_time:5612, sweep_cost_time:273, start_time:"2025-05-31 03:59:27.428642", last_end_time:"2025-05-31 03:59:27.434527", hold_info:nothing, mark_finished:true}, map_cnt=57) [2025-05-31 02:59:27.435052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.445596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.456336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.467022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.477328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.480486] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=41] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:27.482120] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=53] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:27.482299] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=52] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678362234341000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:27.487775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.499440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.509974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.511386] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=4737, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:27.511423] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1159, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:27.511482] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=182] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=30, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:27.511501] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=9264, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:27.511532] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=121, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:27.512216] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:27.524017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.534577] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=31] ts response task statistics(alloc_count=486962, free_count=243480) [2025-05-31 02:59:27.534632] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.534676] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678367033026], gts:1748678367072435000, latest_srr:[mts=1748678367534621]}) [2025-05-31 02:59:27.546486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.558687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.569477] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.579933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.591617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=132][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.591725] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=95] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591732] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591765] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591791] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591799] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.591899] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=58] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:27.592140] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:27.596932] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A76-0-0] [lt=45] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:27.597476] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83100][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38DB-0-0] [lt=53][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525416779647, cache_obj->added_lc()=false, cache_obj->get_object_id()=53714, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:27.597623] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:27.604205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.604390] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A5696-0-0] [lt=21] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:27.612798] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:27.616985] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38DD-0-0] [lt=73][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525416799086, cache_obj->added_lc()=false, cache_obj->get_object_id()=53715, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:27.617143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.627286] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.636352] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:27.637571] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.647715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.659200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.669345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.679549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.689881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.700103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.710332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.720642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.731929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.742095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.753447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.766847] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.769172] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:27.769230] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=49] temporary file module use no blocks [2025-05-31 02:59:27.769244] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:27.769260] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:27.769274] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525415951103, flush_io_finished_ret_:0, flush_io_finished_round_:19935, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:27.788992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.805488] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.806198] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:27.815949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.829553] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.836815] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:27.840818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.852897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.863135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.873467] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.888113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.898520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.908744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.913504] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:244536 disk:44K/10:244536 t1m2:44K/10:244536 t1m2g0:44K/10:244536 |q0: 0 |q1: 0 [2025-05-31 02:59:27.919254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.922596] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:27.923571] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47] get_max_applied_scn(scn={val:1748678367708105000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137429444}, palf_committed_end_scn_:{val:1748678367909551000, v:0}, last_check_scn_:{val:1748678367909551000, v:0}, max_applied_cb_scn_:{val:1748678367708105000, v:0}}) [2025-05-31 02:59:27.923613] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137429444}, readable_log_tail:{lsn:2137429444}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9923132, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:27.931655] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A0-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:27.932024] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.942722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.953284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.963738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.975898] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.976006] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=19] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:27.986635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:27.996846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.009103] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=86] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:28.010103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.018563] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:28.023749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.034328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.035282] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:28.035691] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=79] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678367535008], gts:1748678367572679000, latest_srr:[mts=1748678368035664]}) [2025-05-31 02:59:28.037707] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=111] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:28.045047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.055728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.066082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.075786] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41] LogLoopThread round_cost_time(us)(round_cost_time=20) [2025-05-31 02:59:28.077230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.089496] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A7-0006366543FE0640-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678368087283000, start_id=1748678368087283000, end_id=1748678368087283001) [2025-05-31 02:59:28.089782] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.090358] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=27] gc stale ls task succ [2025-05-31 02:59:28.092566] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:28.099540] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=10] [APPLY STAT CB APPEND COST TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=150, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:28.099584] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=6561, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:28.099605] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=21] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:28.099630] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=24] [APPLY STAT CB EXECUTE TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=-1409, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:28.099638] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38DA-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=47, stat_interval=5000000, avg (count/cost)=5302, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:28.102621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.113086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.113287] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=185] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:28.113336] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [29559][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=38] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:28.115152] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:28.116167] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:28.116219] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=52] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:28.123393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.133811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.144087] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.144192] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678368144188, bytes: 7139122307, bw: 0.270456 MB/s, add_ts: 1002479, add_bytes: 284297 [2025-05-31 02:59:28.154343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.164525] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.167719] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1748678368167714, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007364, add_bytes: 0 [2025-05-31 02:59:28.176811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.187665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.197916] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.205049] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B06-0-0] [lt=20] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:28.208352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.209224] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:28.216433] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=64, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:28.216507] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3981, avg_group_log_size=398) [2025-05-31 02:59:28.216539] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3981) [2025-05-31 02:59:28.216577] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:28.217793] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3981) [2025-05-31 02:59:28.219415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.221738] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32115][T1_L0_G1][T1][YB42AC10871E-0006366549CC1277-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=2003, avg_log_submit_to_slide_time=4418) [2025-05-31 02:59:28.221993] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3981, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:28.230064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.239115] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:28.240631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.250905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.261250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.273129] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.283390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.294478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.294590] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2079-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:28.294677] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A2079-0-0] [lt=76] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=88) [2025-05-31 02:59:28.295407] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5981-0-0] [lt=24] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678363685332, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:28.295455] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5981-0-0] [lt=46] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678363687730, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:28.295481] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81060][TimerWK0_DumpLSLoc][T0][YB42AC1087A6-0006366547BA5981-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678363696536, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678363696543, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:28.305035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.313693] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBA4-0-0] [lt=82] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750417518615722656e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544602828077998765e+03) [2025-05-31 02:59:28.315592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.318565] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678368105662000, v:0}, new_max_version={val:1748678368105662000, v:0}, last_min_version={val:1748678367708105000, v:0}, last_max_version={val:1748678367708105000, v:0}, total_time_us=6585, delta_us=212899) [2025-05-31 02:59:28.318617] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678368105662000, v:0}, min_version={val:1748678368105662000, v:0}, max_version={val:1748678368105662000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:28.326993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.337305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.348652] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA8E-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:28.349158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.359300] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.359390] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.359490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.361793] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.363718] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.375648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.379243] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=33] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:28.379314] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=70] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:28.379699] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F50-0-0] [lt=12] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:28.379742] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F50-0-0] [lt=39] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:28.386008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.396218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.409287] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.413210] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:28.419950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.420472] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [26524][T1_L0_G1][T1][YB42AC10871E-0006366549CC1279-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3981) [2025-05-31 03:59:28.423102] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [32846][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:28.425896] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:28.425966] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44605, [delay/us]:prepare:12, schedule:151, submit:59, rt:1644, total:1867) [2025-05-31 02:59:28.425985] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:28.426051] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44608, [delay/us]: prepare:12, schedule:151, submit:59, device:1644, total:1867) [2025-05-31 02:59:28.426064] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:28.426088] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4055, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:7350, total:7364) [2025-05-31 02:59:28.426134] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:28.426168] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4055, [delay/us]: prepare:0, schedule:0, submit:600000000, device:7350, total:7364) [2025-05-31 02:59:28.426266] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:28.426312] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:28.426322] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:28.426330] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:28.426338] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:28.426347] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=272kB/s, net_out=225kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:28.426364] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] successfully alloc block(macro_id={[ver=1,mode=0,seq=399463][2nd=3068]}) [2025-05-31 02:59:28.427166] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=300] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:28.437358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.440507] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:28.450310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.460749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.471131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.481487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.492276] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.502720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.513090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:28.519013] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC127A-0-0] [lt=26] init point thread id with(&point=0x55c0908c8740, idx_=78, point=[thread id=32846, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=32846) [2025-05-31 02:59:28.523298] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2411, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:28.523409] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=100] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=267, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:28.523492] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:28.523514] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4829, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:28.523545] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=46, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:28.523592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.534052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.545917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.547382] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=64] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:28.557210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.568737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.578966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.589126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.590682] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.590700] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.590769] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.590800] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.593030] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.593100] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.593131] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:28.594269] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:28.602527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.609549] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83090][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38DF-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525417791717, cache_obj->added_lc()=false, cache_obj->get_object_id()=53716, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:28.614011] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:28.615366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.616512] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:28.616550] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:28.616836] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:28.616944] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:28.616978] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:28.624563] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=37] ls blacklist refresh finish(cost_time=42659) [2025-05-31 02:59:28.627751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.638047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.641700] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=104] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:28.650857] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.661210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=125][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.671612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.683744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.688028] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=38] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:28.688090] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81742][T1_TimerWK1_TabletShell][T1][Y0-0000000000000000-0-0] [lt=42] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1028) [2025-05-31 02:59:28.694405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.705642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.718052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.724234] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:28.733745] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DE-0-0] [lt=39] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:28.735590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.747007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.757163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.767533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=161][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.770046] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=54] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:28.770134] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=80] temporary file module use no blocks [2025-05-31 02:59:28.770145] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:28.770211] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:28.770232] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525416951514, flush_io_finished_ret_:0, flush_io_finished_round_:19936, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:28.777775] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.788063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.795308] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10871E-000636654E089A4C-0-0] [lt=27][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525417977476, cache_obj->added_lc()=false, cache_obj->get_object_id()=53717, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:28.800354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.809206] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-05-31 02:59:28.809237] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:59:28.809261] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:28.812391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.814215] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:28.820975] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:36K/8:117054 disk:36K/8:117054 t1m2:36K/8:117054 t1m2g0:36K/8:117054 |q0: 0 |q1: 0 [2025-05-31 02:59:28.822628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.835137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.842256] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:28.845261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.855593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.868092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.878324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.884376] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:28.884516] WDIAG read_multi_row (ob_table_access_helper.h:238) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:28.886460] INFO delete_row (ob_table_access_helper.h:339) [82984][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=128] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678368884645) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678368884645) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:59:28.891233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.902247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.912481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.915879] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] no need to update min/max version(new_version={val:1748678368516055000, v:0}, min_version={val:1748678368516055000, v:0}, max_version={val:1748678368516055000, v:0}) [2025-05-31 02:59:28.923524] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:28.924153] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [33020][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:28.928141] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81742][T1_TimerWK1_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=16] clear dtl interm result cost(us)(clear_cost=3633, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:59:28.931609] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137432836}], scn_array=[cnt:1, {val:1748678368715063002, v:0}], curr_size=122, accum_size=2133, time_cost=12479) [2025-05-31 02:59:28.933685] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A1-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:28.934148] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:28.945982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.956504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.956538] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:59:28.966822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.976173] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=131] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:28.977191] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.987452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:28.997790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.008134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.014479] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:29.019529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.029956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.037906] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=19] get number(ret=0, service_type_=0, range=1, base_id=1748678369036953000, start_id=1748678369036953000, end_id=1748678369036953001) [2025-05-31 02:59:29.037981] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=59] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:29.040536] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.043356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:29.051295] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.060945] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-05-31 02:59:29.060989] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:29.061669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.066684] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [28578][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD43-0-0] [lt=28] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678369065288, keepalive_ts:-1}) [2025-05-31 02:59:29.066738] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [28578][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD43-0-0] [lt=51] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678369065288, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:29.073092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.083334] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.088697] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-05-31 02:59:29.090789] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=24] gc stale ls task succ [2025-05-31 02:59:29.094061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.094698] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:29.105807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.115511] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:29.116391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.116669] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:29.119040] INFO [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:196) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=57] skip repeat study(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678309051820, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, current_timestamp=1748678369119036, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-05-31 02:59:29.121535] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=106] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678152214536000, snapshot_type=1, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:59:29.121587] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=46] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009655406000, snapshot_type=1, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:59:29.121602] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678190211897000, snapshot_type=1, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:59:29.121611] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677990551999001, snapshot_type=1, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:59:29.121620] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185755930000, snapshot_type=1, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:59:29.121628] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307817947000, snapshot_type=1, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:59:29.121636] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678151786499001, snapshot_type=2, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:59:29.121702] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=60] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009054758001, snapshot_type=2, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:59:29.121721] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678189609335001, snapshot_type=2, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:59:29.121751] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=27] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677989766058002, snapshot_type=2, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:59:29.121760] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185265507000, snapshot_type=2, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:59:29.121796] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=32] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307233204000, snapshot_type=2, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:59:29.121905] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=103] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678151908197000, snapshot_type=3, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:59:29.121916] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678009457625000, snapshot_type=3, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:59:29.121924] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678189822651000, snapshot_type=3, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:59:29.121933] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748677990139537000, snapshot_type=3, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:59:29.121941] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678185611745000, snapshot_type=3, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:59:29.121949] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=1748678307718042000, snapshot_type=3, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:59:29.121957] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678152237990, addr="172.16.135.10:2882", gc_status=0) [2025-05-31 02:59:29.121965] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678009656643, addr="172.16.135.166:2882", gc_status=0) [2025-05-31 02:59:29.121973] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678190227064, addr="172.16.135.167:2882", gc_status=0) [2025-05-31 02:59:29.121991] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748677990562207, addr="172.16.135.168:2882", gc_status=0) [2025-05-31 02:59:29.122003] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678185782915, addr="172.16.135.20:2882", gc_status=0) [2025-05-31 02:59:29.122011] INFO [MVCC] collect (ob_multi_version_garbage_collector.cpp:714) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] multi version garbage colloector collects successfully(sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , snapshot_version=4611686018427387903, snapshot_type=4, create_time=1748678307864591, addr="172.16.135.30:2882", gc_status=0) [2025-05-31 02:59:29.122384] INFO [MVCC] decide_reserved_snapshot_version_ (ob_multi_version_garbage_collector.cpp:625) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update global reserved snapshot(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678309051820, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, global_reserved_snapshot_={val:1748677989766058002, v:0}, reserved_snapshot={val:1748677989766058002, v:0}) [2025-05-31 02:59:29.122409] INFO [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:559) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24] multi-version garbage collector refresh successfully(this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678309051820, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:1748677989766058002, v:0}, reserved_snapshot_type:2, reserved_status:-1, reserved_create_time:1748677990562207, reserved_addr:"172.16.135.168:2882", is_this_server_disabled:false, status:0}) [2025-05-31 02:59:29.122427] INFO [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:231) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] repeat refresh successfully(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678369122426, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, current_timestamp=1748678369119146, GARBAGE_COLLECT_RETRY_INTERVAL=60000000) [2025-05-31 02:59:29.122439] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:266) [83017][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] skip repeat reclaim(ret=0, this={this:0x2b090499d030, last_study_timestamp:1748678009672928, last_refresh_timestamp:1748678369122426, last_reclaim_timestamp:1748677829477332, last_sstable_overflow_timestamp:0, has_error_when_study:false, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1748677989766058002, v:0}, is_inited:true}, current_timestamp=1748678369122438, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-05-31 02:59:29.123113] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:29.129324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.142011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.146772] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1748678369146767, bytes: 7139395426, bw: 0.259797 MB/s, add_ts: 1002579, add_bytes: 273119 [2025-05-31 02:59:29.153018] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.163304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.175134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.175562] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1748678369175557, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007843, add_bytes: 0 [2025-05-31 02:59:29.185400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.195737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.206407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=171][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.215607] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:29.216700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.221137] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78] get number(ret=0, service_type_=0, range=1, base_id=1748678369220496000, start_id=1748678369220496000, end_id=1748678369220496001) [2025-05-31 02:59:29.221408] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=93, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:29.221462] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=8, avg_log_batch_cnt=1, total_group_log_size=3310, avg_group_log_size=413) [2025-05-31 02:59:29.221485] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=2843) [2025-05-31 02:59:29.221500] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:29.223432] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3310) [2025-05-31 02:59:29.227007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.227282] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC127F-0-0] [lt=39] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=22, avg_log_gen_to_submit_time=142, avg_log_submit_to_flush_time=5903, avg_log_submit_to_slide_time=6522) [2025-05-31 02:59:29.227501] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3310, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:29.238612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=171][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.244008] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:29.249381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.256504] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=79] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:29.259965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.270479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.282516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.293035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.303324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.312541] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0E42-0-0] [lt=38] handle gts request statistics(total_rt=1133, total_cnt=181, avg_rt=6.225274725274725363e+00, avg_cnt=1.810000000000000142e+01) [2025-05-31 02:59:29.313662] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.320440] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=40] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:29.324465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.337744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.348462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=260][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.358733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.360827] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.360988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.361038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.362210] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.368596] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:455) [81744][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=52] skip defragment task(ret=0, need_defragment=false) [2025-05-31 02:59:29.368623] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:497) [81744][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=25] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=11) [2025-05-31 02:59:29.368636] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:515) [81744][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=12] successfully defragment data blocks(rewrite_cnt=0) [2025-05-31 02:59:29.368646] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:751) [81744][T1_TimerWK3_SSTableDefragme][T1][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=349942, total_skip_cnt=6068, not_in_mem_tablet_cnt=516, invalid_attr_tablet_cnt=0) [2025-05-31 02:59:29.372474] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA8F-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:29.373693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.378559] INFO pn_ratelimit (group.c:653) [81065][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=28] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:29.384418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.395511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.408819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.415936] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:29.419120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=104][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.422550] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBB3-0-0] [lt=8] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750417518615722656e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544602828077998765e+03) [2025-05-31 02:59:29.425043] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=2843) [2025-05-31 02:59:29.430627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=206][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.431034] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678369220496000, v:0}, new_max_version={val:1748678369220496000, v:0}, last_min_version={val:1748678368715063002, v:0}, last_max_version={val:1748678368715063002, v:0}, total_time_us=10296, delta_us=210534) [2025-05-31 02:59:29.431076] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678369220496000, v:0}, min_version={val:1748678369220496000, v:0}, max_version={val:1748678369220496000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:29.435211] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:29.435285] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:8, norm_iops:2, bw:36523, [delay/us]:prepare:12, schedule:554, submit:113, rt:3002, total:3682) [2025-05-31 02:59:29.435304] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:29.435365] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:8, avg_bw:36523, [delay/us]: prepare:12, schedule:554, submit:113, device:3002, total:3682) [2025-05-31 02:59:29.435379] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:29.435406] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4058, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:298, total:310) [2025-05-31 02:59:29.435431] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:29.435459] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4058, [delay/us]: prepare:0, schedule:0, submit:600000000, device:298, total:310) [2025-05-31 02:59:29.435477] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:29.435491] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:29.435500] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:29.435509] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:29.435517] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:29.435525] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=248kB/s, net_out=176kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:29.435545] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399464][2nd=2363]}) [2025-05-31 02:59:29.441626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.444547] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:29.446091] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2025-05-31 02:59:29.447030] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=48] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:59:29.451938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.462321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.475316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.485581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.498640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.509268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.519684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.525484] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=4058, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:29.525530] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=800, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:29.525593] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=36, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:29.527028] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC1282-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=16, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:29.527056] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC1282-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=6079, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:29.530766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.538834] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=65] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678369538759], gts:1748678369537499000, latest_srr:[mts=1748678369538759]}) [2025-05-31 02:59:29.539012] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=132] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:29.541828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.551938] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=38] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:29.552265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.566515] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.577047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.587430] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.590621] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.590951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.591010] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.591190] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.592105] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.592715] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=47] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.592825] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:29.595858] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=24] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:29.597812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.608207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.617309] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=519024, replace_num=15728) [2025-05-31 02:59:29.617392] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:29.619483] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38E1-0-0] [lt=38][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525418801652, cache_obj->added_lc()=false, cache_obj->get_object_id()=53718, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:29.622044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.623610] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=144] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:29.632937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.643212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.646590] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:29.653441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.663694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.676549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.687210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.697484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.701533] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F7E3ECF-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1748678369698296000, start_id=1748678369698296000, end_id=1748678369698296001) [2025-05-31 02:59:29.707741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.718643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.724323] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678369622937000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678369724263}, version_delta=101383) [2025-05-31 02:59:29.728851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.741180] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.751151] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678359745519, end_time=1748678369751146, dag_status={}) [2025-05-31 02:59:29.751196] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=40] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751212] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751219] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751227] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751234] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751241] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751248] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751255] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751262] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:29.751272] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:59:29.751281] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:59:29.751455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.761826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.772050] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:29.772092] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] temporary file module use no blocks [2025-05-31 02:59:29.772103] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:29.772177] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=70] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:29.772193] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525417952446, flush_io_finished_ret_:0, flush_io_finished_round_:19937, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:29.772394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=126][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.782544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.792857] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.797284] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83067][T1_L5_G0][T1][YB42AC108714-000636671ECD5CCE-0-0] [lt=30] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=285, Holding Memory(MB)=134, Tenant Remain Memory(MB)=251, Usable Remain Memory(MB)=150, Last Update Limit Timestamp=1748678369794525, Is Updated=true) [2025-05-31 02:59:29.804262] WDIAG [STORAGE.TRANS] inner_lock_for_read (ob_tx_data_functor.cpp:338) [83074][T1_L0_G0][T1][YB42AC10871E-0006366542D8A743-0-0] [lt=46][errcode=-6004] lock_for_read need retry(ret=-6004, tx_data={tx_id:{txid:11578144}, ref_cnt:1, state:"RUNNING", commit_version:{val:1748678369794525000, v:0}, start_scn:{val:1748678369794525000, v:0}, end_scn:{val:18446744073709551615, v:3}, op_guard:{tx_data_op_:null}}, lock_for_read_arg_={mvcc_acc_ctx_:{type:1, abs_lock_timeout_ts:1748678399701778, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678369794525000, v:0}, tx_id:{txid:11578145}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, data_trans_id_:{txid:11578144}, data_sql_sequence_:{branch:0, seq:3}, read_latest_:false, read_uncommitted_:false, scn_:{val:1748678369794525000, v:0}}, tx_cc_ctx={state:10, prepare_version:{val:1748678369794525000, v:0}}) [2025-05-31 02:59:29.804818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.805794] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83074][T1_L0_G0][T1][YB42AC10871E-0006366542D8A743-0-0] [lt=87] get number(ret=0, service_type_=0, range=1, base_id=1748678369805278000, start_id=1748678369805278000, end_id=1748678369805278001) [2025-05-31 02:59:29.815454] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.820089] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:29.827072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.836153] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:29.840919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.847411] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:29.851253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.867345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.882605] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.884503] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=37] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19331174400, fast_delete_log_mode=false) [2025-05-31 02:59:29.896333] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.904157] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] BlockGCTimerTask success(ret=0, cost_time_us=23, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:59:29.906817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.917428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.928748] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:76, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:184046323036, recv_hp_rpc_cnt:74963, recv_np_rpc_cnt:879130, recv_lp_rpc_cnt:0, recv_mysql_cnt:47155, recv_task_cnt:2476, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1234211 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9890, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:544614, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1748678368923736}{group_id:10, queue_size:0, recv_req_cnt:104091, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18390, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678362495292}{group_id:2, queue_size:0, recv_req_cnt:151107, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101246, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:108074, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:272917, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=697 pcode=0x717:cnt=488 pcode=0x1502:cnt=244 pcode=0x710:cnt=161 pcode=0x523:cnt=97, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:59:29.930236] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1485] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:82822978, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:59:29.931250] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=817] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84382256, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:59:29.932392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.933044] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:400252 disk:52K/12:400252 t1m2:52K/12:400252 t1m2g0:52K/12:400252 |q0: 0 |q1: 0 [2025-05-31 02:59:29.936903] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A2-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:29.942599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.942649] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:29.944756] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [32115][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=47] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:29.952933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.963386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.973763] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.976376] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=31] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:29.984199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:29.994577] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.004836] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.016729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=133][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.021243] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:30.026954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.037127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:30.038281] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [33405][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:30.038538] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 03:59:30.039234] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [33405][T1_L0_G1][T1][YB42AC1087A7-00063665517F56C3-0-0] [lt=26] init point thread id with(&point=0x55c0908d1300, idx_=637, point=[thread id=33405, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=33405) [2025-05-31 02:59:30.039828] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:30.048660] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:30.048961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.061847] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.066918] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=34] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:30.072280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=153][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.082603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.089965] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2025-05-31 02:59:30.091038] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=41] gc stale ls task succ [2025-05-31 02:59:30.094651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.096549] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=130] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:30.104871] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.115925] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=55] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:30.117298] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=39] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:30.118073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.120340] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81743][T1_TimerWK2_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=69] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:30.124158] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:30.128061] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=110] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678369823798000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=304251, in_cluster_service=true, cluster_version={val:1748678369823798000, v:0}, min_cluster_version={val:1748678369823798000, v:0}, max_cluster_version={val:1748678369823798000, v:0}, get_cluster_version_err=0, cluster_version_delta=304251, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195540, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=361073291, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:30.128381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.138626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.147019] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1748678370147015, bytes: 7139534333, bw: 0.132439 MB/s, add_ts: 1000248, add_bytes: 138907 [2025-05-31 02:59:30.149144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.160366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.170795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.182934] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1748678370182929, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007372, add_bytes: 0 [2025-05-31 02:59:30.183388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.193669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.205691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.216560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.221646] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:30.224079] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4913) [2025-05-31 02:59:30.227598] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4913, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:30.227839] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [33020][T1_L0_G1][T1][YB42AC1087A7-00063665517F56C4-0-0] [lt=24] init point thread id with(&point=0x55c0908cb2c0, idx_=252, point=[thread id=33020, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=33020) [2025-05-31 02:59:30.228025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.228738] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=121] no need to update min/max version(new_version={val:1748678369823798000, v:0}, min_version={val:1748678369823798000, v:0}, max_version={val:1748678369823798000, v:0}) [2025-05-31 02:59:30.238272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.251956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.252374] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:30.262532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.272892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.283822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.294044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.304297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.309673] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207A-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:30.309714] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207A-0-0] [lt=30] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=41) [2025-05-31 02:59:30.316057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.326405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.331293] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=74, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:30.331333] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5035, avg_group_log_size=419) [2025-05-31 02:59:30.331354] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5502) [2025-05-31 02:59:30.331369] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:30.332304] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:140) [81744][T1_TimerWK3_CompactionRefre][T1][YB42AC1087A6-00063665431A60F1-0-0] [lt=13] InfoPoolResizeTask(cost_ts=10) [2025-05-31 02:59:30.333671] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=112, avg_log_submit_to_flush_time=4738, avg_log_submit_to_slide_time=4900) [2025-05-31 02:59:30.337043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.349578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.358923] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=127] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.359001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.359030] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.359787] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.360082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.376513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.380395] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=43] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:30.380420] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=24] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:30.380472] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F51-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:30.380491] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F51-0-0] [lt=15] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:30.390337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.392566] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA90-0-0] [lt=55][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:30.399960] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=1997, gc_interval=10000000) [2025-05-31 02:59:30.400017] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=33] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:59:30.400039] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] gc_check_member_list_ cost time(ret=0, time_us=45) [2025-05-31 02:59:30.400052] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:59:30.400063] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:59:30.400076] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:59:30.400087] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=12) [2025-05-31 02:59:30.400095] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] ObSafeDestroyHandler start process [2025-05-31 02:59:30.400103] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:59:30.400179] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=62] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:59:30.400669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.411092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.421321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.423215] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=99] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:30.434169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.444400] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:30.444400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.444517] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4468, iops:11, norm_iops:2, bw:48704, [delay/us]:prepare:12, schedule:226, submit:519, rt:2488, total:3245) [2025-05-31 02:59:30.444540] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:30.444601] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4468, avg_iops:11, avg_bw:48702, [delay/us]: prepare:12, schedule:226, submit:519, device:2488, total:3245) [2025-05-31 02:59:30.444615] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:30.444642] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4058, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3135, total:3528) [2025-05-31 02:59:30.444667] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:30.444695] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4058, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3135, total:3528) [2025-05-31 02:59:30.444714] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:30.444729] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:30.444737] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:30.444746] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:30.444754] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:30.444763] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=137kB/s, net_out=95kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:30.444786] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399465][2nd=2645]}) [2025-05-31 02:59:30.452985] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:30.455119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.467453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.477707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.490291] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=63] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:30.490947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.492070] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=51] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:30.492193] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678362234341000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:30.501226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.511479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.522542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.534551] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBC0-0-0] [lt=29] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.702417564392089844e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.630610169395342382e+03) [2025-05-31 02:59:30.535270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.536113] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC128D-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5501) [2025-05-31 02:59:30.536773] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3827, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:30.536872] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=91] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=921, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:30.536982] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:30.536998] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5271, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:30.537086] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:30.540750] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=102] ts response task statistics(alloc_count=487034, free_count=243516) [2025-05-31 02:59:30.540785] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=726, get_gts_with_stc_cnt=30, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=29, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:30.540830] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678370039933], gts:1748678370078422001, latest_srr:[mts=1748678370540802]}) [2025-05-31 02:59:30.542905] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678370216445001, v:0}, new_max_version={val:1748678370216445001, v:0}, last_min_version={val:1748678369823798001, v:0}, last_max_version={val:1748678369823798001, v:0}, total_time_us=11525, delta_us=326457) [2025-05-31 02:59:30.543011] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=99] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678370216445001, v:0}, min_version={val:1748678370216445001, v:0}, max_version={val:1748678370216445001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:30.545497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.557342] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA8F-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525419739511, cache_obj->added_lc()=false, cache_obj->get_object_id()=53719, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:30.557640] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.557766] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] ====== checkpoint timer task ====== [2025-05-31 02:59:30.558509] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=18] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:30.558582] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=72] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:30.559003] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=27] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:30.559093] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:30.559152] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:30.559167] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678363778615001, v:0}) [2025-05-31 02:59:30.559175] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:30.559182] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:30.559197] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:30.559216] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:30.559229] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:30.559249] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:30.559271] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:30.559281] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81743][T1_TimerWK2_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:30.569623] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81742][T1_TimerWK1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=66] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:30.569715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.580172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.587072] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A77-0-0] [lt=28] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:30.590516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.590736] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.591009] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.591269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.591304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.592214] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.593581] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.593640] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:30.597984] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:30.603501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.616322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.624778] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:30.628124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:30.629320] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [33609][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:30.629374] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread created(tenant_->id()=1, group_id_=19, token=2) [2025-05-31 02:59:30.629489] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38E3-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525419811658, cache_obj->added_lc()=false, cache_obj->get_object_id()=53720, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:30.629713] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:30.641095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.642100] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38E5-0-0] [lt=91][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525419824269, cache_obj->added_lc()=false, cache_obj->get_object_id()=53721, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:30.643578] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38E7-0-0] [lt=24][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:30.644642] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38E9-0-0] [lt=23][errcode=-4201] hashtable not init [2025-05-31 02:59:30.651385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.654649] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:30.661645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.671883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.681445] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC10870A-0006366715B99E37-0-0] [lt=45] get number(ret=0, service_type_=0, range=1, base_id=1748678370680196000, start_id=1748678370680196000, end_id=1748678370680196001) [2025-05-31 02:59:30.683007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.693219] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.703449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.714056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.717384] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83101][T1_L0_G0][T1][YB42AC108714-000636671E6D514A-0-0] [lt=266][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525419899470, cache_obj->added_lc()=false, cache_obj->get_object_id()=53722, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:30.725182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.736041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.741516] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20DF-0-0] [lt=37] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:30.746826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.749572] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137441035}], scn_array=[cnt:1, {val:1748678370731994000, v:0}], curr_size=589, accum_size=2576, time_cost=14030) [2025-05-31 02:59:30.750252] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=217746, log_service_ts=1748678370532186, min_tx_service_ts=4611686018427387, timestamp={val:1748678370532186000, v:0}) [2025-05-31 02:59:30.750285] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=58, Other_Error_Cnt=0, Submit_Succ_Cnt=41, last_scn="{val:1748678366024257000, v:0}", last_lsn={lsn:2137422120}, last_gts={val:1748678370731994000, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-31 02:59:30.757298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.767592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.772394] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:30.772449] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-05-31 02:59:30.772462] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:30.772479] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:30.772496] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525418954407, flush_io_finished_ret_:0, flush_io_finished_round_:19938, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:30.778073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.788272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.799631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.799904] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:792) [81065][TimerWK3_AutoLSLocRpc][T0][YB42AC1087A6-0006366545EA56F0-0-0] [lt=28] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.16.135.166:2882", "172.16.135.167:2882", "172.16.135.30:2882", "172.16.135.10:2882", "172.16.135.168:2882", "172.16.135.20:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}}, {key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}}, {key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, location:{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}}]) [2025-05-31 02:59:30.809879] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.810996] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1085) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=57] replay status is not follower(min_unreplayed_lsn={lsn:1979154949}, base_lsn={lsn:1543409664}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137441624}, readable_log_tail:{lsn:2137441624}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9910952, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:30.811462] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=463] get_replay_process success(id={id:1}, submitted_log_size=435745285, unsubmitted_log_size=0, replayed_log_size=435745285, unreplayed_log_size=0) [2025-05-31 02:59:30.811502] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [81742][T1_TimerWK1_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=16] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=415, replayed_log_size(MB)=415, last_submitted_log_size(MB)=415, last_replayed_log_size(MB)=415, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-31 02:59:30.820198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.822090] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1F35-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1748678370820000000, start_id=1748678370820000000, end_id=1748678370820000001) [2025-05-31 02:59:30.822619] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-05-31 02:59:30.822653] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:59:30.822733] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=69] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:30.825040] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:30.830493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.840890] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.851122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.855324] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:30.864053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=186][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.874227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.880160] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=39][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:30.884564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.894895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.905212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.912497] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81060][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=41] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:30.915645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.926480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.937082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.939511] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=133] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:787865 disk:48K/11:787865 t1m2:48K/11:787865 t1m2g0:48K/11:787865 |q0: 0 |q1: 0 [2025-05-31 02:59:30.939772] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A3-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:30.947685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.948681] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:30.960410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.971125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.976588] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=57] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:30.980425] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=42] ObArchiveSender is running(thread_index=0) [2025-05-31 02:59:30.981421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:30.992126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.002341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.011764] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", [SlidingWindow]={palf_id:1, self:"172.16.135.166:2882", lsn_allocator:{max_log_id:4032658, max_lsn:2137442334, max_scn:{val:1748678370930413000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:1971066992}, reuse_lsn:{lsn:2137442334}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:2095499294}}, last_submit_lsn:{lsn:2137441746}, last_submit_end_lsn:{lsn:2137442334}, last_submit_log_id:4032658, last_submit_log_pid:11, max_flushed_lsn:{lsn:2137441746}, max_flushed_end_lsn:{lsn:2137442334}, max_flushed_log_pid:11, committed_end_lsn:{lsn:2137442334}, last_slide_log_id:4032658, last_slide_scn:{val:1748678370930413000, v:0}, last_slide_lsn:{lsn:2137441746}, last_slide_end_lsn:{lsn:2137442334}, last_slide_log_pid:11, last_slide_log_accum_checksum:3664802787, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b09131e4530}, [StateMgr]={this:0x2b09131e89b0, self:"172.16.135.166:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, leader:"172.16.135.166:2882", leader_epoch:2, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:4032659, is_changing_config_with_arb:false, reconfirm_start_time_us:1748659848751813, palf_role_change_cb:0x2b09131ef470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.16.135.166:2882", alive_paxos_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, reconfig_barrier:{prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_end_lsn:{lsn:1979154949}, prev_mode_pid:11}, persistent_config_version:{proposal_id:11, config_seq:15}, ms_ack_list:0, resend_config_version:{proposal_id:11, config_seq:15}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1748659847838949, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1748659847838949, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b09131e5c70}, [ModeMgr]={palf_id:1, self:"172.16.135.166:2882", applied_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1748659848952559, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1748315799404644001, v:0}, min_block_id:8, min_block_min_scn:{val:1748253949550669001, v:0}, base_lsn_for_block_gc:{lsn:2013143040}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:11}, log_config_meta:{version:3, proposal_id:11, prev:{version:1, config:{config_version:{proposal_id:10, config_seq:14}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:11, config_seq:15}, log_sync_memberlist:3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:10, prev_lsn:{lsn:1979153877}, prev_mode_pid:11}, log_snapshot_meta:{version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:11, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:200704}, readable_log_tail:{lsn:200704}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2147352576}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66904064, block_header_serialize_buf_:0x2b09131eabb8, flashback_version:0}, log_storage:{log_tail:{lsn:2137442334}, readable_log_tail:{lsn:2137442334}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9910242, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1748659848751996, this:0x2b09131e90f0}, [Reconfirm]={palf_id:1, self:"172.16.135.166:2882", state:"FINISHED", new_proposal_id:11, prepare_log_ack_list:2"172.16.135.167:2882";"172.16.135.30:2882";, curr_paxos_follower_list:2{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.16.135.166:2882", majority_max_accept_pid:10, majority_max_lsn:{lsn:1979154949}, saved_end_lsn:{lsn:1979154949}, last_submit_prepare_req_time_us:1748659848752358, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b09131e8b30}) [2025-05-31 02:59:31.012100] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=325] [PALF_DUMP](palf_id=1, self="172.16.135.166:2882", ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678370948662, last_flushed_end_lsn:{lsn:2137442334}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678370944495, last_flushed_end_lsn:{lsn:2137442334}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678370946979, last_flushed_end_lsn:{lsn:2137442334}}]) [2025-05-31 02:59:31.012128] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] current log_task status(palf_id=1, self="172.16.135.166:2882", log_id=4032659, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}, last_ack_time_us:1748678370948662, last_flushed_end_lsn:{lsn:2137442334}}, {member:{server:"172.16.135.166:2882", timestamp:1, flag:0}, last_ack_time_us:1748678370944495, last_flushed_end_lsn:{lsn:2137442334}}, {member:{server:"172.16.135.167:2882", timestamp:1, flag:0}, last_ack_time_us:1748678370946979, last_flushed_end_lsn:{lsn:2137442334}}]) [2025-05-31 02:59:31.012984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.023516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.025232] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:31.035298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.044319] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:31.044675] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:31.044698] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678370541255], gts:1748678370579173000, latest_srr:[mts=1748678371044596]}) [2025-05-31 02:59:31.046637] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.046684] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:31.046798] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [31689][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=39] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:31.055781] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:31.056956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.062580] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] ====== check clog disk timer task ====== [2025-05-31 02:59:31.062628] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81743][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=40] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:31.068121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.078367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.082239] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [81940][T1_RFLWorker][T1][YB42AC1087A6-0006366541E9FC95-0-0] [lt=24] ObRemoteFetchWorker is running(thread_index=0) [2025-05-31 02:59:31.088983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.091298] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=42] gc stale ls task succ [2025-05-31 02:59:31.099666] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:31.099826] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.110312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.116184] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:31.117787] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:31.122325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.129923] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:31.131321] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B08-0-0] [lt=49] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:31.132731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.142175] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:31.142219] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] try gc retain ctx [2025-05-31 02:59:31.142228] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-31 02:59:31.142300] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1748678370930413000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678361008386000, v:0}, keep_alive_scn_:{val:1748678366024257000, v:0}, update_ts_:1748678371142166}) [2025-05-31 02:59:31.143495] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=84] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:31.144047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.151459] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1748678371151430, bytes: 7139680299, bw: 0.138592 MB/s, add_ts: 1004415, add_bytes: 145966 [2025-05-31 02:59:31.154301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.157995] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [32846][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA6-0-0] [lt=28] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678371191495, keepalive_ts:-1}) [2025-05-31 02:59:31.158032] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [32846][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA6-0-0] [lt=35] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678371191495, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:31.167306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.177792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.180654] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=20] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:31.180722] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=25] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:31.180742] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81744][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=1046) [2025-05-31 02:59:31.189554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.190550] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1748678371190545, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-31 02:59:31.200309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.212427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.223154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.226619] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:31.231612] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=40] === Run print tenant memory usage task === [2025-05-31 02:59:31.231726] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=37] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 120,934,400 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 647,544,832 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,585,280 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,748,678,371,231,513 active_memstore_used= 132,120,576 total_memstore_used= 132,120,576 total_memstore_hold= 132,120,576 memstore_freeze_trigger_limit= 53,228,740 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 891,678,720 max_mem_memstore_can_get_now= 266,143,726 memstore_alloc_pos= 291,504,128 memstore_frozen_pos= 159,383,552 memstore_reclaimed_pos= 159,383,552 [2025-05-31 02:59:31.232154] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13] [CHUNK_MGR] limit= 6,442,450,944 hold= 3,073,150,976 total_hold= 3,323,985,920 used= 1,570,394,112 freelists_hold= 1,502,756,864 total_maps= 2,019 total_unmaps= 807 large_maps= 150 large_unmaps= 0 huge_maps= 45 huge_unmaps= 44 resident_size= 0 unmanaged_memory_size= 120,934,400 virtual_memory_used= 4,344,475,648 [OS_PARAMS] vm.max_map_count= 655,360 transparent_hugepages= never [CHUNK_MGR] 2 MB_CACHE: hold= 1,008,730,112 count= 481 pushes= 14,535,468 pops= 14,534,987 maps= 1,824 unmaps= 763 [CHUNK_MGR] 4 MB_CACHE: hold= 134,844,416 count= 58 pushes= 151,415 pops= 151,357 maps= 87 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 48,599,040 count= 9 pushes= 85 pops= 76 maps= 20 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 22,954 pops= 22,950 maps= 5 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 95,682,560 count= 11 pushes= 38 pops= 27 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 24,035,328 count= 2 pushes= 2 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 31,236,096 count= 2 pushes= 2 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 134,381,568 count= 8 pushes= 16 pops= 8 maps= 11 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-05-31 02:59:31.232187] INFO print (ob_malloc_time_monitor.cpp:37) [81063][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-31 02:59:31.233401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.246529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=159][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.247531] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4240) [2025-05-31 02:59:31.247712] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4240, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:31.257435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.257582] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:31.267975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.278362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.289321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.300999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.311256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.321812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.332105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.341953] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=74, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:31.341998] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=4707, avg_group_log_size=392) [2025-05-31 02:59:31.342020] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=4240) [2025-05-31 02:59:31.342092] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:31.342680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.345534] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=402, avg_log_gen_to_submit_time=493, avg_log_submit_to_flush_time=6508, avg_log_submit_to_slide_time=6855) [2025-05-31 02:59:31.355970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:31.356860] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [33875][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:31.358866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=111] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.359119] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=55] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.359156] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.359183] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.360136] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:31.375085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.385999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.397751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.408073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.418308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.422552] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA91-0-0] [lt=34][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:31.428698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=135][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.429476] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:31.438919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:31.448525] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [33875][T1_L0_G1][T1][YB42AC10871E-0006366549CC1298-0-0] [lt=426] init point thread id with(&point=0x55c0908d8880, idx_=1107, point=[thread id=33875, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=33875) [2025-05-31 02:59:31.449787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.454824] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:31.454992] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=128] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, norm_iops:3, bw:52693, [delay/us]:prepare:16, schedule:735, submit:690, rt:4151, total:5595) [2025-05-31 02:59:31.455015] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:31.455145] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=113] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52692, [delay/us]: prepare:16, schedule:735, submit:690, device:4151, total:5595) [2025-05-31 02:59:31.455161] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:31.455487] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=308] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4052, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:401, total:480) [2025-05-31 02:59:31.455624] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:31.455670] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4051, [delay/us]: prepare:0, schedule:0, submit:600000000, device:401, total:480) [2025-05-31 02:59:31.455691] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:31.455707] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:31.455743] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:31.455754] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:31.455763] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:31.455773] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=155kB/s, net_out=104kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:31.455795] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399466][2nd=2927]}) [2025-05-31 02:59:31.460058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.460371] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:31.461948] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2025-05-31 02:59:31.470324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.480597] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.490757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.496135] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [30576][T1_L0_G19][T1][YB42AC10871E-00063665511850DB-0-0] [lt=133] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.30:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:31.496185] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [30576][T1_L0_G19][T1][YB42AC10871E-00063665511850DB-0-0] [lt=38] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.30:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:31.500925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.511242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.522269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.533093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.541678] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBCD-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.726416969299316406e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.587160814026798562e+03) [2025-05-31 02:59:31.543450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.545314] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC1299-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=4241) [2025-05-31 02:59:31.546344] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=5914, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:31.546377] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=376, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:31.546422] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=10, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:31.546438] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=7494, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:31.546457] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=65, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:31.546737] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=122] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678371137961002, v:0}, new_max_version={val:1748678371137961002, v:0}, last_min_version={val:1748678371137961001, v:0}, last_max_version={val:1748678371137961001, v:0}, total_time_us=6779, delta_us=408772) [2025-05-31 02:59:31.546771] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678371137961002, v:0}, min_version={val:1748678371137961002, v:0}, max_version={val:1748678371137961002, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:31.557364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.567721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.577516] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=9] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:31.580205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.582269] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=127] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:31.584331] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=83] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:31.586451] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=47] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:31.586501] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=48] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:31.588363] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:31.588459] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=47] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:31.588484] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=23] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=1, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:31.588510] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:31.589112] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=597] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:31.589156] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=35] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:31.589168] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:31.589191] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:31.589211] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:31.590530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.590638] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.590679] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.594951] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.595016] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.595044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.595120] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.595190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:31.599622] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:31.599679] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=52] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:31.599692] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=11] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:31.599736] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:31.599751] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=14] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:31.599760] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:31.599773] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:31.599790] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE1-0-0] [lt=15] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:31.600632] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:31.600935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.613917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.614492] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:31.614648] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.614809] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=124][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.614937] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.624977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.629603] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:31.630506] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=73] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:31.636503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.636603] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2025-05-31 02:59:31.639448] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [30576][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=29] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-05-31 02:59:31.639608] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13] start to print blacklist info [2025-05-31 02:59:31.639829] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=104] new mysql sessid created(conn.sessid_=3221740601, support_ssl=false) [2025-05-31 02:59:31.639955] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=88] sm conn init succ(conn.sessid_=3221740601, sess.client_addr_="127.0.0.1:49033") [2025-05-31 02:59:31.639975] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x2b08ff896450, session_id:3221740601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:141}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:31.640012] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=22] ls blacklist refresh finish(cost_time=49680) [2025-05-31 02:59:31.640823] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-31 02:59:31.640936] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [81569][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=81] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-31 02:59:31.641033] INFO load_privilege_info (obmp_connect.cpp:564) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFE0-0-0] [lt=58] no tenant name set, use default tenant name(tenant_name=sys) [2025-05-31 02:59:31.642333] INFO [SERVER] process (obmp_connect.cpp:504) [82988][T1_MysqlQueueTh][T1][Y0-000636654429FFE0-0-0] [lt=17] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=%, sessid=3221740601, proxy_sessid=0, client_sessid=3221740601, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=true, proc_ret=0, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-31 02:59:31.646453] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83077][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38EB-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525420828622, cache_obj->added_lc()=false, cache_obj->get_object_id()=53723, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.646873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.648830] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38ED-0-0] [lt=23][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525420831002, cache_obj->added_lc()=false, cache_obj->get_object_id()=53724, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.651803] WDIAG [RPC.OBMYSQL] handle_sock_event (ob_sql_nio.cpp:1005) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] socket closed, it maybe disconnected by the client or by observer actively(mask=8197, *s={this:0x2b08ff896450, session_id:3221740601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678371647078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:159, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:31.651948] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=136] kill and revert session(conn.sessid_=3221740601, proxy_sessid=0, ret=0) [2025-05-31 02:59:31.651964] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:980) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] sock ref clean, do destroy(*s={this:0x2b08ff896450, session_id:3221740601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:false, reader:{fd:141}, err:5, last_decode_time:1748678371647078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:159, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-31 02:59:31.652006] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] unlock session of tenant(conn.sessid_=3221740601, proxy_sessid=0, conn.tenant_id_=1) [2025-05-31 02:59:31.652018] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [81569][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221740601, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42AC1087A6-000636654C3A38EE-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:14, pkt_rec[0]:{send:obp_mysql_header_{len_:11, seq_:3}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[1]:{send:obp_mysql_header_{len_:11, seq_:2}, pkt_name:"PKT_OKP", obp_mysql_header_.is_send_:1}, pkt_rec[2]:{receive:obp_mysql_header_{len_:15, rec_:15, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[3]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[4]:{send:obp_mysql_header_{pkt_num_:1, seq_:3}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[5]:{send:obp_mysql_header_{len_:9, seq_:4}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[6]:{send:obp_mysql_header_{pkt_num_:9, seq_:13}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[7]:{send:obp_mysql_header_{len_:9, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[8]:{receive:obp_mysql_header_{len_:26, rec_:26, seq_:0}, pkt_name:"COM_QUERY", obp_mysql_header_.type_:3, obp_mysql_header_.is_send_:0}, pkt_rec[9]:{send:obp_mysql_header_{len_:5, seq_:2}, pkt_name:"PKT_RESHEAD", obp_mysql_header_.is_send_:1}, pkt_rec[10]:{send:obp_mysql_header_{pkt_num_:11, seq_:13}, pkt_name:"PKT_FIELD", obp_mysql_header_.is_send_:1}, pkt_rec[11]:{send:obp_mysql_header_{len_:9, seq_:14}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}, pkt_rec[12]:{send:obp_mysql_header_{pkt_num_:6, seq_:20}, pkt_name:"PKT_ROW", obp_mysql_header_.is_send_:1}, pkt_rec[13]:{send:obp_mysql_header_{len_:9, seq_:21}, pkt_name:"PKT_EOF", obp_mysql_header_.is_send_:1}], disconnect_state=4) [2025-05-31 02:59:31.652232] INFO [SERVER] run (obmp_disconnect.cpp:78) [83089][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=44] free session successfully(sessid=3221740601) [2025-05-31 02:59:31.652253] INFO [SERVER] run (obmp_disconnect.cpp:85) [83089][T1_L0_G0][T1][Y0-0000000000000000-0-0] [lt=20] mark session id unused(sessid=3221740601) [2025-05-31 02:59:31.657322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.662793] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:31.667623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.679261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.684073] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81743][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=14] ====== traversal_flush timer task ====== [2025-05-31 02:59:31.684544] INFO [MDS] flush (mds_table_mgr.cpp:184) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF7-0-0] [lt=28] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678371540557000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678380364653000, latest_log_ts:{val:1748678363778615001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:31.684837] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81743][T1_TimerWK2_Flush][T1][YB42AC1087A6-00063665430A5EF7-0-0] [lt=285] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:31.689470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.699722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.712732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.722932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.733247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.743514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.746118] INFO [PALF] inner_write_once_ (log_block_handler.cpp:403) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=57200876, this={dio_aligned_buf:{buf_write_offset:825, buf_padding_size:3271, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:390771, truncate_used_ts:132366}, log_block_size:67108864, io_fd:{first_id:-1, second_id:125, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2b08efdc0080}}, aligned_buf_len=4096, aligned_block_offset=57200640, buf_len=589, total_write_size=47330, total_write_size_after_dio=520192, ob_pwrite_used_ts=445459, count=115, each_pwrite_cost=3873) [2025-05-31 02:59:31.753760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.764481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.772667] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:31.772708] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-31 02:59:31.772718] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:31.772746] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:31.772763] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525419954712, flush_io_finished_ret_:0, flush_io_finished_round_:19939, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:31.774725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.782922] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:31.785555] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81744][T1_TimerWK3_ReqMemEvict][T1][YB42AC1087A6-00063665431A60F2-0-0] [lt=29] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=31) [2025-05-31 02:59:31.785791] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678371785097000, start_id=1748678371785097000, end_id=1748678371785097001) [2025-05-31 02:59:31.786452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.797077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.802306] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=18] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:31.802342] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=29] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:31.802356] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:31.802367] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:31.805195] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=314, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53694, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:31.806004] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81744][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=805] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:31.807311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.809328] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC10871E-000636654DFA0E51-0-0] [lt=70] get number(ret=0, service_type_=0, range=1, base_id=1748678371806240000, start_id=1748678371806240000, end_id=1748678371806240001) [2025-05-31 02:59:31.817818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.828023] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.833503] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=534752, replace_num=15728) [2025-05-31 02:59:31.833541] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:31.838369] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=140][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.849137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.859374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.863326] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:31.869748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.880580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=270][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.892388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.902616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.912933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.923263] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=138][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.940597] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3F8-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525421122767, cache_obj->added_lc()=false, cache_obj->get_object_id()=53725, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:31.941204] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=184][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.949280] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A4-0-0] [lt=44][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:31.950024] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:40K/9:173684 disk:40K/9:173684 t1m2:40K/9:173684 t1m2g0:40K/9:173684 |q0: 0 |q1: 0 [2025-05-31 02:59:31.951604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.962012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.962148] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137445725}], scn_array=[cnt:1, {val:1748678371942028000, v:0}], curr_size=588, accum_size=3025, time_cost=12448) [2025-05-31 02:59:31.972400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.976780] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=30] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:31.982773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:31.994079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.007150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.017522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.027729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.033743] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:32.037912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.047780] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:32.048767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.059086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.064355] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=58] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:32.069552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.079790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.090110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.090688] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=48] current monitor number(seq_=-1) [2025-05-31 02:59:32.091451] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=100] gc stale ls task succ [2025-05-31 02:59:32.102047] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:32.102318] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.112541] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.116457] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:32.120238] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=30] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:32.122774] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.132027] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=88] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:32.133092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.143787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.151553] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=290] PNIO [ratelimit] time: 1748678372151549, bytes: 7139869474, bw: 0.180390 MB/s, add_ts: 1000119, add_bytes: 189175 [2025-05-31 02:59:32.153813] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:32.154051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.164925] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.175173] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.179308] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81429][pnio1][T0][YB42AC10870A-000636671AB947F5-0-0] [lt=11][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:1713141692, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678372212880, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1391493, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678372203029, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678372179303, enqueue_timestamp_:0, request_arrival_time_:1748678372179303, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:32.179384] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81429][pnio1][T0][YB42AC10870A-000636671AB947F5-0-0] [lt=59][errcode=-5150] deliver rpc request fail(&req=0x2b0919e04098, ret=-5150) [2025-05-31 02:59:32.185508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.192336] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678372192331, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001786, add_bytes: 0 [2025-05-31 02:59:32.195851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.207096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.217457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.228392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.233916] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:32.235662] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=35] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:32.238657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.249253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.253745] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3858) [2025-05-31 02:59:32.258792] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3858, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:32.261354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.264994] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=58] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:32.271636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.281963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.292267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.302630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.313002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.324747] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207B-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:32.324784] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207B-0-0] [lt=27] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=38) [2025-05-31 02:59:32.325112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.335744] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.346278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.348464] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=113] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=111, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:32.348507] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=5190, avg_group_log_size=471) [2025-05-31 02:59:32.348528] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5190) [2025-05-31 02:59:32.348542] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:32.352437] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12A2-0-0] [lt=42] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=16, avg_log_gen_to_submit_time=144, avg_log_submit_to_flush_time=3323, avg_log_submit_to_slide_time=5186) [2025-05-31 02:59:32.356559] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.356616] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:32.358905] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.358944] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [28578][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=18] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:32.361387] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.361459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.361527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.367776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.381084] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=32] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:32.381899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.385807] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=30] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:32.385834] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81065][TimerWK3_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:32.386251] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F52-0-0] [lt=10] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:32.386306] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F52-0-0] [lt=51] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:32.392792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=285][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.403638] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.416562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.427109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.439739] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:32.439794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.447795] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA92-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:32.450502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.459199] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33E2-0-0] [lt=95][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525421641370, cache_obj->added_lc()=false, cache_obj->get_object_id()=53726, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:32.460657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.467871] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=176] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:32.467939] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4840, iops:11, norm_iops:3, bw:52567, [delay/us]:prepare:12, schedule:91, submit:53, rt:2728, total:2886) [2025-05-31 02:59:32.467957] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:32.468013] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4841, avg_iops:11, avg_bw:52571, [delay/us]: prepare:12, schedule:91, submit:53, device:2728, total:2886) [2025-05-31 02:59:32.468027] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:32.468052] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4045, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:331, total:346) [2025-05-31 02:59:32.468077] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:32.468105] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4045, [delay/us]: prepare:0, schedule:0, submit:600000000, device:331, total:346) [2025-05-31 02:59:32.468122] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:32.468136] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:32.468145] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:32.468153] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:32.468161] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:32.468170] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=193kB/s, net_out=161kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:32.468191] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399467][2nd=2222]}) [2025-05-31 02:59:32.468805] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:32.472393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.484723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.495754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.506224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.519224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.530392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.542712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.549394] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678372549329], gts:1748678372547794000, latest_srr:[mts=1748678372549329]}) [2025-05-31 02:59:32.550088] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBDC-0-0] [lt=7] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750568914413452148e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544337160538230364e+03) [2025-05-31 02:59:32.554029] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12A4-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5190) [2025-05-31 02:59:32.554114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=658][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.557264] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3203, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:32.557315] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=687, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:32.558957] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=15, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:32.558982] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5825, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:32.559003] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:32.559469] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678372261251001, v:0}, new_max_version={val:1748678372261251001, v:0}, last_min_version={val:1748678372047726000, v:0}, last_max_version={val:1748678372047726000, v:0}, total_time_us=11216, delta_us=298215) [2025-05-31 02:59:32.559506] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678372261251001, v:0}, min_version={val:1748678372261251001, v:0}, max_version={val:1748678372261251001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:32.564457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.574713] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.585470] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.591294] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591343] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591418] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591446] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.591538] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=37] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:32.593001] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:32.595798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.603698] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:32.604652] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81063][TimerWK1_ServerTracerTim][T0][YB42AC1087A6-00063665443A569F-0-0] [lt=34] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:32.608304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.618489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.629034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.632255] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:32.639387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.640269] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:32.649696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.659393] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F0-0-0] [lt=96][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525421841560, cache_obj->added_lc()=false, cache_obj->get_object_id()=53727, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:32.660287] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.669905] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:32.671157] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.683113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.694927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.705383] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.716052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.729549] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.739799] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.749970] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20E0-0-0] [lt=17] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:32.752809] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.763153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.774918] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2716) [33609][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1B1-0-0] [lt=119] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"172.16.135.167:2882", ls_array_:[{id:1}]}) [2025-05-31 02:59:32.774982] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1629) [33609][T1_L0_G19][T1][YB42AC1087A7-00063665477DB1B1-0-0] [lt=53] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"172.16.135.30:2882", timestamp:1748657046111340, flag:0}{server:"172.16.135.166:2882", timestamp:1, flag:0}{server:"172.16.135.167:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="172.16.135.167:2882", this={role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-05-31 02:59:32.775521] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:32.775556] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=27] temporary file module use no blocks [2025-05-31 02:59:32.775567] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:32.775680] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:32.775700] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525420954979, flush_io_finished_ret_:0, flush_io_finished_round_:19940, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:32.775755] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.786136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=177][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.796749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.807847] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=165][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.819222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=482][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.829419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.836437] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2334) [83067][T1_L5_G0][T1][YB42AC10870A-000636672468BDE9-0-0] [lt=35] need check schema(new_schema_version=1748674258961328, cached_tenant_schema_version=1748674258961328, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT start_service_time FROM __all_server WHERE svr_ip = ? AND svr_port = ?) [2025-05-31 02:59:32.840170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.840978] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:32.849008] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=10] ====== tenant freeze timer task ====== [2025-05-31 02:59:32.849046] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=27] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:59:32.849070] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:32.850812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.861160] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=139][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.872666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.873364] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:32.882949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.893380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.903685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.916054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.926567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.936811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.947593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.951404] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A5-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:32.958010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.966071] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=99] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:56K/13:153211 disk:56K/13:153211 t1m2:56K/13:153211 t1m2g0:56K/13:153211 |q0: 0 |q1: 0 [2025-05-31 02:59:32.967823] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get_max_applied_scn(scn={val:1748678372754497000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137451200}, palf_committed_end_scn_:{val:1748678372955201000, v:0}, last_check_scn_:{val:1748678372955201000, v:0}, max_applied_cb_scn_:{val:1748678372754497000, v:0}}) [2025-05-31 02:59:32.967946] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=113] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137451200}, readable_log_tail:{lsn:2137451200}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9901376, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:32.968256] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=34] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:32.968980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.976970] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:32.979201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.989444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:32.999675] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=29] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:32.999729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.010049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.020439] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.032110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.041214] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:33.041383] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:33.041419] INFO [COMMON] handle (memory_dump.cpp:531) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=34] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-31 02:59:33.042043] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] update_check_range(min_check_version=1986, max_check_version=1987, global_version=1988) [2025-05-31 02:59:33.052063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.052706] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678373041371000, start_id=1748678373041371000, end_id=1748678373041371001) [2025-05-31 02:59:33.064526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.065143] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-05-31 02:59:33.065171] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:33.073153] INFO handle (memory_dump.cpp:658) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 149 275 0 1 2 63 1 0 1 4 1 1 0 1 5 23 6 0 1 7 1 2 0 1 8 52 1 0 1 12 17 3 0 1 13 3 1 0 1 14 1 1 0 1 15 1 1 0 1 18 7 3 0 1 20 2 1 0 500 0 88 252 0 500 7 63 5 0 500 8 36 1 0 500 9 2 1 0 500 10 3 2 0 500 16 1 1 0 500 17 10 7 0 500 22 39 63 0 500 23 20 10 0 508 0 2 7 0 508 8 5 1 0 509 0 2 7 0 509 8 5 1 0 cost_time: 31135 [2025-05-31 02:59:33.073203] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24] [MEMORY] tenant: 1, limit: 1,073,741,824 hold: 891,678,720 cache_hold: 85,983,232 cache_used: 85,983,232 cache_item_count: 41 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 447,062,016 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 48,234,496 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 109,051,904 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 35,651,584 limit= 214,748,360 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PS_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 14,700,544 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:33.073742] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 447,062,016 used= 335,946,736 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 32 [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31,428,608 used= 31,368,256 count= 7 avg_used= 4,481,179 block_cnt= 7 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 29,130,752 used= 29,116,416 count= 14 avg_used= 2,079,744 block_cnt= 14 chunk_cnt= 14 mod=CACHE_MAP_NODE [MEMORY] hold= 16,797,696 used= 16,777,336 count= 1 avg_used= 16,777,336 block_cnt= 1 chunk_cnt= 1 mod=MrgMemPoolChk [MEMORY] hold= 16,232,912 used= 16,067,217 count= 111 avg_used= 144,749 block_cnt= 39 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 12,075,008 used= 11,858,496 count= 28 avg_used= 423,517 block_cnt= 28 chunk_cnt= 16 mod=MysqlRequesReco [MEMORY] hold= 11,497,472 used= 11,461,632 count= 4 avg_used= 2,865,408 block_cnt= 4 chunk_cnt= 3 mod=ServerObjecPool [MEMORY] hold= 10,510,336 used= 10,491,392 count= 1 avg_used= 10,491,392 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 9,510,912 used= 9,437,472 count= 9 avg_used= 1,048,608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9,142,272 used= 9,050,112 count= 12 avg_used= 754,176 block_cnt= 12 chunk_cnt= 11 mod=ResourceGroup [MEMORY] hold= 8,777,728 used= 8,760,064 count= 2 avg_used= 4,380,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,409,088 used= 8,388,640 count= 1 avg_used= 8,388,640 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 7,274,496 used= 6,389,760 count= 108 avg_used= 59,164 block_cnt= 108 chunk_cnt= 28 mod=[T]char [MEMORY] hold= 5,386,240 used= 5,368,320 count= 1 avg_used= 5,368,320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3,865,600 used= 3,705,600 count= 800 avg_used= 4,632 block_cnt= 800 chunk_cnt= 13 mod=CkptDgnMem [MEMORY] hold= 3,772,416 used= 3,751,944 count= 1 avg_used= 3,751,944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3,723,264 used= 3,698,704 count= 2 avg_used= 1,849,352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3,211,264 used= 3,131,136 count= 14 avg_used= 223,652 block_cnt= 14 chunk_cnt= 11 mod=DiagnosticSum [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 5 mod=SqlDtlQueue [MEMORY] hold= 3,047,424 used= 3,013,468 count= 10 avg_used= 301,346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3,002,368 used= 2,982,720 count= 1 avg_used= 2,982,720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,818,048 used= 2,380,480 count= 86 avg_used= 27,680 block_cnt= 86 chunk_cnt= 25 mod=DiagnosticInfo [MEMORY] hold= 2,486,272 used= 2,457,920 count= 2 avg_used= 1,228,960 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,816 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,126,336 used= 2,036,832 count= 448 avg_used= 4,546 block_cnt= 436 chunk_cnt= 34 mod=WorkerMap [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1,474,560 used= 1,310,720 count= 20 avg_used= 65,536 block_cnt= 20 chunk_cnt= 16 mod=MajorMergeMgr [MEMORY] hold= 1,402,496 used= 1,380,008 count= 76 avg_used= 18,158 block_cnt= 76 chunk_cnt= 3 mod=TabletMap [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsCache [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPsInfo [MEMORY] hold= 1,015,808 used= 498,976 count= 62 avg_used= 8,048 block_cnt= 62 chunk_cnt= 22 mod=[T]ObPerfEventR [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829,632 used= 821,248 count= 2 avg_used= 410,624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 798,112 used= 789,624 count= 3 avg_used= 263,208 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 786,688 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 737,280 used= 655,360 count= 10 avg_used= 65,536 block_cnt= 10 chunk_cnt= 7 mod=MinorMergeMgr [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 650,240 used= 634,880 count= 80 avg_used= 7,936 block_cnt= 80 chunk_cnt= 14 mod=TltMemtablMgr [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 532,480 used= 531,072 count= 1 avg_used= 531,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 503,936 used= 492,032 count= 62 avg_used= 7,936 block_cnt= 62 chunk_cnt= 18 mod=LinearHashMapNo [MEMORY] hold= 491,520 used= 468,576 count= 3 avg_used= 156,192 block_cnt= 3 chunk_cnt= 3 mod=SQLSessionInfo [MEMORY] hold= 475,136 used= 458,816 count= 2 avg_used= 229,408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 458,752 used= 268,800 count= 28 avg_used= 9,600 block_cnt= 28 chunk_cnt= 15 mod=[T]ObDSActionAr [MEMORY] hold= 422,784 used= 412,672 count= 51 avg_used= 8,091 block_cnt= 51 chunk_cnt= 18 mod=ParamCharsType [MEMORY] hold= 416,768 used= 399,872 count= 88 avg_used= 4,544 block_cnt= 88 chunk_cnt= 23 mod=[T]ObTraceEvent [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 401,408 used= 393,504 count= 1 avg_used= 393,504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 369,680 count= 2 avg_used= 184,840 block_cnt= 2 chunk_cnt= 2 mod=SqlPlanMon [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 311,104 used= 303,904 count= 6 avg_used= 50,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 286,720 used= 280,000 count= 1 avg_used= 280,000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 278,464 used= 270,468 count= 2 avg_used= 135,234 block_cnt= 2 chunk_cnt= 2 mod=CsSstableReader [MEMORY] hold= 246,992 used= 3,318 count= 3,061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212,992 used= 211,072 count= 1 avg_used= 211,072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 211,328 used= 206,336 count= 26 avg_used= 7,936 block_cnt= 26 chunk_cnt= 14 mod=QueryAllocator [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,328 used= 141,880 count= 3 avg_used= 47,293 block_cnt= 3 chunk_cnt= 3 mod=CommSysVarFac [MEMORY] hold= 131,072 used= 130,560 count= 2 avg_used= 65,280 block_cnt= 2 chunk_cnt= 2 mod=MemTblObj [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 2 mod=TabletIDSetBkt [MEMORY] hold= 122,688 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=DbmsSched_Job [MEMORY] hold= 121,920 used= 119,040 count= 15 avg_used= 7,936 block_cnt= 15 chunk_cnt= 10 mod=CtxBlockPl [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 106,496 used= 98,320 count= 1 avg_used= 98,320 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 100,736 used= 26,128 count= 370 avg_used= 70 block_cnt= 98 chunk_cnt= 25 mod=Coro [MEMORY] hold= 98,304 used= 91,968 count= 1 avg_used= 91,968 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 94,752 count= 1 avg_used= 94,752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 88,000 used= 85,888 count= 11 avg_used= 7,808 block_cnt= 11 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 81,920 used= 73,896 count= 1 avg_used= 73,896 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairCache [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81,824 used= 73,472 count= 2 avg_used= 36,736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 80,000 used= 78,080 count= 10 avg_used= 7,808 block_cnt= 10 chunk_cnt= 7 mod=HashNodeLCSta [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,280 count= 1 avg_used= 65,280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,408 used= 63,488 count= 4 avg_used= 15,872 block_cnt= 4 chunk_cnt= 3 mod=IOUsageInfo [MEMORY] hold= 65,024 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=ColEqualPl [MEMORY] hold= 57,344 used= 51,328 count= 1 avg_used= 51,328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 57,088 used= 52,776 count= 3 avg_used= 17,592 block_cnt= 3 chunk_cnt= 3 mod=HB_SERVICE [MEMORY] hold= 57,024 used= 44,352 count= 66 avg_used= 672 block_cnt= 61 chunk_cnt= 22 mod=[T]ObWarningBuf [MEMORY] hold= 56,896 used= 55,552 count= 7 avg_used= 7,936 block_cnt= 7 chunk_cnt= 7 mod=DictPl [MEMORY] hold= 51,072 used= 48,384 count= 14 avg_used= 3,456 block_cnt= 10 chunk_cnt= 9 mod=CACHE_INST [MEMORY] hold= 49,152 used= 42,496 count= 1 avg_used= 42,496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48,960 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 48,880 used= 44,760 count= 2 avg_used= 22,380 block_cnt= 2 chunk_cnt= 2 mod=Autoincrement [MEMORY] hold= 48,768 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 6 mod=IntDiffPl [MEMORY] hold= 48,000 used= 33,600 count= 75 avg_used= 448 block_cnt= 44 chunk_cnt= 12 mod=OMT_Worker [MEMORY] hold= 45,328 used= 4,824 count= 201 avg_used= 24 block_cnt= 75 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 4 mod=MdsTableMgr [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=ConstPl [MEMORY] hold= 40,640 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 5 mod=RawPl [MEMORY] hold= 35,584 used= 30,400 count= 3 avg_used= 10,133 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_AssignedMgr [MEMORY] hold= 32,704 used= 25,191 count= 2 avg_used= 12,595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 32,000 used= 31,232 count= 4 avg_used= 7,808 block_cnt= 4 chunk_cnt= 4 mod=HashNodePlanCac [MEMORY] hold= 24,960 used= 23,808 count= 6 avg_used= 3,968 block_cnt= 6 chunk_cnt= 3 mod=CkpDgn [MEMORY] hold= 24,576 used= 16,392 count= 1 avg_used= 16,392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,384 used= 20,152 count= 2 avg_used= 10,076 block_cnt= 2 chunk_cnt= 2 mod=PlGdbHelper [MEMORY] hold= 24,384 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=RlePl [MEMORY] hold= 24,368 used= 20,136 count= 2 avg_used= 10,068 block_cnt= 2 chunk_cnt= 2 mod=TabletStats [MEMORY] hold= 24,336 used= 20,104 count= 2 avg_used= 10,052 block_cnt= 2 chunk_cnt= 2 mod=TLD_ResourceMgr [MEMORY] hold= 23,952 used= 23,376 count= 3 avg_used= 7,792 block_cnt= 3 chunk_cnt= 1 mod=CkptDgnMemNode [MEMORY] hold= 17,024 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 8,200 count= 1 avg_used= 8,200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 9,792 count= 1 avg_used= 9,792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16,384 used= 13,896 count= 1 avg_used= 13,896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16,384 used= 14,016 count= 1 avg_used= 14,016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSession [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=HexStrPl [MEMORY] hold= 16,256 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=StrDiffPl [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 12,864 used= 12,288 count= 3 avg_used= 4,096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 10,496 used= 10,112 count= 2 avg_used= 5,056 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 9,712 used= 9,312 count= 2 avg_used= 4,656 block_cnt= 2 chunk_cnt= 2 mod=HashBucPsSessIn [MEMORY] hold= 9,456 used= 9,064 count= 2 avg_used= 4,532 block_cnt= 2 chunk_cnt= 2 mod=GROUP_INDEX_MAP [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LsElectRefInfo [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompPairs [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FillCache [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DbmsScheduler [MEMORY] hold= 8,096 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=PsSessionInfo [MEMORY] hold= 8,000 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsInfo [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=CkptDgnCUNode [MEMORY] hold= 7,984 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,920 used= 7,728 count= 1 avg_used= 7,728 block_cnt= 1 chunk_cnt= 1 mod=HashNodePsCache [MEMORY] hold= 7,904 used= 7,712 count= 1 avg_used= 7,712 block_cnt= 1 chunk_cnt= 1 mod=DmlStatNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CheckMapNode [MEMORY] hold= 7,824 used= 7,632 count= 1 avg_used= 7,632 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoNode [MEMORY] hold= 7,552 used= 7,352 count= 1 avg_used= 7,352 block_cnt= 1 chunk_cnt= 1 mod=mrg_prog_cker [MEMORY] hold= 6,912 used= 6,288 count= 3 avg_used= 2,096 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 6,816 used= 5,600 count= 7 avg_used= 800 block_cnt= 7 chunk_cnt= 4 mod=DICollector [MEMORY] hold= 5,056 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,832 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4,288 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 3,520 count= 4 avg_used= 880 block_cnt= 2 chunk_cnt= 2 mod=PoolArenaArray [MEMORY] hold= 3,328 used= 1,792 count= 8 avg_used= 224 block_cnt= 2 chunk_cnt= 2 mod=LogIOTask [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3,312 used= 3,112 count= 1 avg_used= 3,112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3,200 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,880 used= 1,920 count= 5 avg_used= 384 block_cnt= 5 chunk_cnt= 4 mod=TableLockNode [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,528 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2,048 used= 1,856 count= 1 avg_used= 1,856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2,016 used= 1,824 count= 1 avg_used= 1,824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,920 used= 1,728 count= 1 avg_used= 1,728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,744 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,472 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1,280 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,248 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,232 used= 1,032 count= 1 avg_used= 1,032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1,200 used= 1,000 count= 1 avg_used= 1,000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1,120 used= 120 count= 5 avg_used= 24 block_cnt= 2 chunk_cnt= 2 mod=FreezeTask [MEMORY] hold= 1,024 used= 832 count= 1 avg_used= 832 block_cnt= 1 chunk_cnt= 1 mod=tenant_rm_mgr [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 672 used= 288 count= 2 avg_used= 144 block_cnt= 2 chunk_cnt= 2 mod=ObTxLSLogCb [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceMap [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 512 used= 120 count= 2 avg_used= 60 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 1 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 400 used= 208 count= 1 avg_used= 208 block_cnt= 1 chunk_cnt= 1 mod=[T]ObMacroBlock [MEMORY] hold= 304 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 48 count= 1 avg_used= 48 block_cnt= 1 chunk_cnt= 1 mod=[T]TSITabletTab [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=[T]TSILastOper [MEMORY] hold= 335,946,736 used= 327,881,787 count= 32,596 avg_used= 10,058 mod=SUMMARY [2025-05-31 02:59:33.073807] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=51] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 132,120,576 used= 131,088,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 block_cnt= 63 chunk_cnt= 63 mod=Memstore [MEMORY] hold= 131,088,384 used= 131,028,408 count= 63 avg_used= 2,079,816 mod=SUMMARY [2025-05-31 02:59:33.073825] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 466,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466,944 used= 460,416 count= 1 avg_used= 460,416 mod=SUMMARY [2025-05-31 02:59:33.073912] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 48,234,496 used= 38,595,328 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,093,824 used= 28,264,124 count= 2,128 avg_used= 13,282 block_cnt= 2,128 chunk_cnt= 23 mod=SqlPhyPlan [MEMORY] hold= 3,001,536 used= 2,779,392 count= 248 avg_used= 11,207 block_cnt= 248 chunk_cnt= 23 mod=SqlPlanCache [MEMORY] hold= 1,605,568 used= 1,480,237 count= 168 avg_used= 8,810 block_cnt= 168 chunk_cnt= 3 mod=SqlPhyPlObj [MEMORY] hold= 1,528,064 used= 1,491,968 count= 188 avg_used= 7,936 block_cnt= 188 chunk_cnt= 15 mod=OB_SQLSTAT_OBJ [MEMORY] hold= 1,364,352 used= 1,243,968 count= 627 avg_used= 1,984 block_cnt= 209 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1,984 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 38,595,328 used= 35,261,289 count= 3,361 avg_used= 10,491 mod=SUMMARY [2025-05-31 02:59:33.073938] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,384 used= 58,751 count= 259 avg_used= 226 block_cnt= 22 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,480 used= 58,783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-31 02:59:33.073952] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 109,051,904 used= 105,283,584 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 block_cnt= 204 chunk_cnt= 52 mod=CoStack [MEMORY] hold= 105,283,584 used= 105,087,744 count= 204 avg_used= 515,136 mod=SUMMARY [2025-05-31 02:59:33.073988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=20] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 85,983,232 used= 85,983,232 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 85,983,232 used= 0 count= 41 avg_used= 0 mod=SUMMARY [2025-05-31 02:59:33.074005] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 35,651,584 used= 34,234,304 limit= 214,748,360 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 29,491,200 used= 26,236,400 count= 400 avg_used= 65,591 block_cnt= 400 chunk_cnt= 15 mod=L_TabletPool [MEMORY] hold= 4,653,056 used= 4,634,880 count= 71 avg_used= 65,280 block_cnt= 71 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 90,048 used= 84,768 count= 2 avg_used= 42,384 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 34,234,304 used= 30,956,048 count= 473 avg_used= 65,446 mod=SUMMARY [2025-05-31 02:59:33.074018] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 6,291,456 used= 3,998,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 block_cnt= 492 chunk_cnt= 3 mod=MemtableCallbac [MEMORY] hold= 3,998,976 used= 3,904,512 count= 492 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:33.074032] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 8,128 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LobPersist [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:33.074045] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= PS_CACHE_CTX_ID hold= 2,097,152 used= 32,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 block_cnt= 7 chunk_cnt= 1 mod=SqlPsCache [MEMORY] hold= 32,528 used= 28,177 count= 22 avg_used= 1,280 mod=SUMMARY [2025-05-31 02:59:33.074068] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 14,700,544 used= 9,050,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5,860,288 used= 5,721,856 count= 721 avg_used= 7,936 block_cnt= 721 chunk_cnt= 5 mod=TX_DATA_SLICE [MEMORY] hold= 3,182,528 used= 3,153,728 count= 3 avg_used= 1,051,242 block_cnt= 3 chunk_cnt= 3 mod=MEMTABLE_ARENA [MEMORY] hold= 8,128 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 9,050,944 used= 8,883,520 count= 725 avg_used= 12,253 mod=SUMMARY [2025-05-31 02:59:33.074084] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 4,194,304 used= 2,893,568 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 block_cnt= 356 chunk_cnt= 2 mod=MdsTable [MEMORY] hold= 2,893,568 used= 2,825,216 count= 356 avg_used= 7,936 mod=SUMMARY [2025-05-31 02:59:33.074119] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 649,641,984 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 220,737,536 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 132,120,576 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 75,497,472 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 8,388,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 26,357,760 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 86,790,144 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 93,458,432 limit= 9,223,372,036,854,775,807 [2025-05-31 02:59:33.074239] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:33.074532] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [81268][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 220,737,536 used= 180,819,920 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 33,292,288 used= 33,275,904 count= 16 avg_used= 2,079,744 block_cnt= 16 chunk_cnt= 16 mod=ObTxDesc [MEMORY] hold= 12,873,728 used= 12,856,400 count= 1 avg_used= 12,856,400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 8,933,376 used= 8,912,896 count= 1 avg_used= 8,912,896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7,274,928 used= 6,334,546 count= 1,107 avg_used= 5,722 block_cnt= 573 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5,824,512 used= 5,792,960 count= 6 avg_used= 965,493 block_cnt= 6 chunk_cnt= 5 mod=IoControl [MEMORY] hold= 5,373,952 used= 5,236,736 count= 32 avg_used= 163,648 block_cnt= 32 chunk_cnt= 11 mod=Iteratorget_tenant_id()=1, queue_size=524288, config_mem_limit_=32212254, request_manager_->get_size_used()=0, evict_high_size_level=471859, evict_low_size_level=419430, allocator->allocated()=2079744, evict_high_mem_level=16106127, evict_low_mem_level=0) [2025-05-31 02:59:33.104118] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:33.111201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.114322] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=48, stat_interval=5000000, avg (count/cost)=165, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:33.114366] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=48, stat_interval=5000000, avg (count/cost)=7032, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:33.114378] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=12] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=48, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:33.114385] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=48, stat_interval=5000000, avg (count/cost)=-913, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:33.114392] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=48, stat_interval=5000000, avg (count/cost)=6284, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:33.117664] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:33.120969] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:33.121389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.128217] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31] LogLoopThread round_cost_time(us)(round_cost_time=21) [2025-05-31 02:59:33.131739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.132362] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:33.141945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.152138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.152621] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=138] PNIO [ratelimit] time: 1748678373152615, bytes: 7140097140, bw: 0.216888 MB/s, add_ts: 1001066, add_bytes: 227666 [2025-05-31 02:59:33.163299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=120][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.173596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.183876] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.194741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.197602] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1748678373197597, bytes: 0, bw: 0.000000 MB/s, add_ts: 1005266, add_bytes: 0 [2025-05-31 02:59:33.205177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.217053] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=127][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.227718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.240098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.241374] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:33.250818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.261145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.271513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=185][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.272162] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5313) [2025-05-31 02:59:33.272265] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5313, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:33.275185] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:33.282091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.293683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.304301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.314655] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.325208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.335761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.347112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=152][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.357420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.357482] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:33.357532] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [27077][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:33.359027] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.360983] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.361024] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.361053] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.361488] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=119] get number(ret=0, service_type_=0, range=1, base_id=1748678373360559000, start_id=1748678373360559000, end_id=1748678373360559001) [2025-05-31 02:59:33.361652] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=82, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:33.361682] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3981, avg_group_log_size=398) [2025-05-31 02:59:33.361701] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3981) [2025-05-31 02:59:33.361716] INFO [LIB] stat (utility.h:1211) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:33.366427] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=6219, avg_log_submit_to_slide_time=6357) [2025-05-31 02:59:33.366957] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:33.368111] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.384043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.396405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.406872] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=180][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.419097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.429454] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.440054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.444031] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:33.450626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.452481] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B42-0-0] [lt=46] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.461992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.474265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.475711] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:33.477232] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2025-05-31 03:59:33.477765] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [34598][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:33.478004] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:33.479575] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA93-0-0] [lt=57][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:33.480014] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:33.480060] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44516, [delay/us]:prepare:14, schedule:259, submit:908, rt:2921, total:4103) [2025-05-31 02:59:33.480076] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:33.480130] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44516, [delay/us]: prepare:14, schedule:259, submit:908, device:2921, total:4103) [2025-05-31 02:59:33.480143] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:33.480168] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4046, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:353, total:370) [2025-05-31 02:59:33.480192] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:33.480220] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4046, [delay/us]: prepare:0, schedule:0, submit:600000000, device:353, total:370) [2025-05-31 02:59:33.480291] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:33.480311] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:33.480320] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:33.480328] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:33.480336] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:33.480345] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=193kB/s, net_out=108kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:33.480366] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399468][2nd=2799]}) [2025-05-31 02:59:33.488213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.502258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.503725] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=26] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:33.505456] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=54] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:33.505598] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=51] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678372256477000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:33.512579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.525546] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.536764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.549026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.553792] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=55] ts response task statistics(alloc_count=487106, free_count=243552) [2025-05-31 02:59:33.553907] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=101] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:33.554088] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678373052799], gts:1748678373091114000, latest_srr:[mts=1748678373553951]}) [2025-05-31 02:59:33.559556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.562151] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFBEB-0-0] [lt=34] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750568914413452148e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544337160538230364e+03) [2025-05-31 02:59:33.565979] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12AE-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3981) [2025-05-31 02:59:33.569057] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4490, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:33.569097] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1486, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:33.569260] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:33.569285] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6408, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:33.569364] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=59, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:33.572365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=77][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:33.572520] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [34598][T1_L0_G1][T1][YB42AC1087A7-00063665517F56E8-0-0] [lt=297] init point thread id with(&point=0x55c0908e3d40, idx_=1830, point=[thread id=34598, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=34598) [2025-05-31 02:59:33.572808] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678373157106000, v:0}, new_max_version={val:1748678373157106000, v:0}, last_min_version={val:1748678373105959000, v:0}, last_max_version={val:1748678373105959000, v:0}, total_time_us=13007, delta_us=415699) [2025-05-31 02:59:33.572842] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678373157106000, v:0}, min_version={val:1748678373157106000, v:0}, max_version={val:1748678373157106000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 03:59:33.573319] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [34633][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:33.573816] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:33.585463] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.588689] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A78-0-0] [lt=23] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:33.590290] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.590336] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.590364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.591197] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.591229] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.591607] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.592093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:33.598402] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B43-0-0] [lt=80] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.598906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.604349] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:33.609312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.619559] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.630337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.632479] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B44-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.632697] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:33.640746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.644792] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:33.651504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.662898] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.671592] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F2-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525422853761, cache_obj->added_lc()=false, cache_obj->get_object_id()=53728, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:33.674216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.680957] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:33.685103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.690002] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B45-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.690430] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=12] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:33.690484] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=37] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=988) [2025-05-31 02:59:33.699715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.711044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.721281] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.732005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.736676] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:149) [38166][T1_L0_G5][T1][YB42AC1087A7-00063665469F7C56-0-0] [lt=95] submit log success(service_type_=0, last_id=1748678390364653000, limited_id=1748678390364653000) [2025-05-31 02:59:33.736758] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:56) [38166][T1_L0_G5][T1][YB42AC1087A7-00063665469F7C56-0-0] [lt=80] submit log with lock success(service_type_=0, last_id=1748678390364653000, limited_id=1748678390364653000) [2025-05-31 02:59:33.739414] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:846) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=7] update id meta(ret=0, service_type=0, limited_id=1748678390364653000, latest_scn={val:1748678373736140001, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}) [2025-05-31 02:59:33.739491] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:181) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=75] handle submit callback(service_type_=0, success=true, used_ts=2817, log_ts={val:1748678373736140001, v:0}, limited_id=1748678390364653000) [2025-05-31 02:59:33.739507] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:555) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=14] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2025-05-31 02:59:33.739517] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:593) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38EA-0-0] [lt=9] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2025-05-31 02:59:33.742248] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.753175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.763707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.773941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.777568] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B46-0-0] [lt=56] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.786105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.796410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.806980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.817254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.821047] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC10871E-000636654E089A50-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525423003216, cache_obj->added_lc()=false, cache_obj->get_object_id()=53729, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:33.827727] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.838071] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.845121] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:33.848243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.858649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.868952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.880123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.881504] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=65] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:33.891506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.899771] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B47-0-0] [lt=29] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.903736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.913954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.924118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.934307] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.935927] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [82996][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=41] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-31 02:59:33.939611] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B48-0-0] [lt=36] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:33.946360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.954097] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A6-0-0] [lt=41][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:33.957316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.967934] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.977205] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:33.978172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.988389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:33.999664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.010078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.023652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.033897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.044135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.046642] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=550480, replace_num=15728) [2025-05-31 02:59:34.046677] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:34.056315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.064349] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678373554476], gts:1748678373593786000, latest_srr:[mts=1748678374064303]}) [2025-05-31 02:59:34.067056] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.077639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.077952] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=40] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:651383 disk:48K/11:651383 t1m2:48K/11:651383 t1m2g0:48K/11:651383 |q0: 0 |q1: 0 [2025-05-31 02:59:34.082394] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:34.088091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.091143] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2025-05-31 02:59:34.093315] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:59:34.099456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.106202] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:34.110695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.117926] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:34.121165] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:34.121400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.131657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.133153] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:34.142125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.153000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.162935] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678374162931, bytes: 7140255943, bw: 0.149900 MB/s, add_ts: 1010316, add_bytes: 158803 [2025-05-31 02:59:34.163636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.174220] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B0A-0-0] [lt=43] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:34.174798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.184082] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=10] finish to resize log external storage handler(current_ts=1748678374184076, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-31 02:59:34.184124] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [81939][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=40] snapshot_traffic_info once [2025-05-31 02:59:34.185671] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=84][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.196232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.205265] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678374205261, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007664, add_bytes: 0 [2025-05-31 02:59:34.207345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.217636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.228348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.240874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.249074] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=81] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:34.251365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.258825] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=101] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:34.261731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.272099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.276755] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3561) [2025-05-31 02:59:34.279626] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3561, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:34.282993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:34.283390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.293947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.304291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.316750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.327226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.327820] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=46] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:34.340098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.341326] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207C-0-0] [lt=28] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:34.341382] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207C-0-0] [lt=45] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=58) [2025-05-31 02:59:34.350580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.361344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.361503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.361546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.361638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.361675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=97] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.365682] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=114] no need to update min/max version(new_version={val:1748678373759638001, v:0}, min_version={val:1748678373759638001, v:0}, max_version={val:1748678373759638001, v:0}) [2025-05-31 02:59:34.371624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.377361] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=137] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=110, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:34.377402] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=3094, avg_group_log_size=309) [2025-05-31 02:59:34.377453] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3561) [2025-05-31 02:59:34.377469] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:34.382561] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12B6-0-0] [lt=33] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=20, avg_log_gen_to_submit_time=149, avg_log_submit_to_flush_time=5568, avg_log_submit_to_slide_time=5861) [2025-05-31 02:59:34.384662] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81743][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=100] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:34.386964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.388111] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:34.388133] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=21] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:34.388223] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F53-0-0] [lt=16] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:34.388246] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F53-0-0] [lt=19] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:34.399090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.409532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.419804] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.431998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.442261] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.450667] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:34.453085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.453296] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=45] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-31 02:59:34.463254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.473666] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.484994] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:34.485122] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, norm_iops:2, bw:44829, [delay/us]:prepare:17, schedule:969, submit:140, rt:2221, total:3349) [2025-05-31 02:59:34.485142] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:34.485201] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44828, [delay/us]: prepare:17, schedule:969, submit:140, device:2221, total:3349) [2025-05-31 02:59:34.485215] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:34.485242] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4075, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:267, total:278) [2025-05-31 02:59:34.485266] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:34.485295] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4075, [delay/us]: prepare:0, schedule:0, submit:600000000, device:267, total:278) [2025-05-31 02:59:34.485313] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:34.485327] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:34.485336] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:34.485344] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:34.485353] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:34.485361] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=161kB/s, net_out=94kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:34.485382] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399469][2nd=2376]}) [2025-05-31 02:59:34.486300] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=64] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:34.486471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.497464] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA94-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:34.497945] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.508076] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.514405] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B49-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:34.518365] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.528956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.539272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.549930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.555253] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:206) [83020][T1_DBMSSched][T1][YB42AC1087A6-0006366551BA2B4A-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-05-31 02:59:34.557135] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:670) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=47] failed to get result(ret=-4008) [2025-05-31 02:59:34.557332] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:515) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=47] check new jobs(ret=0, tenant_id=1, is_oracle_tenant=false, job_infos=[{tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748678246784954, start_date_:1748169746138009, last_date_:1748678246141150, this_date_:0, next_date_:1748679146138009, end_date_:64060560000000000, total_:239131641, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748613666964441, start_date_:1748613600000000, last_date_:1748613600006788, this_date_:0, next_date_:1749218400000000, end_date_:64060560000000000, total_:66957653, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748314024043185, start_date_:1748268000000000, last_date_:0, this_date_:0, next_date_:1748872800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748600847610427, start_date_:1748255246137533, last_date_:1748600846140058, this_date_:0, next_date_:1748687246137533, end_date_:64060560000000000, total_:3757845, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748168846136670, start_date_:1748700000000000, last_date_:0, this_date_:0, next_date_:1748700000000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748624400066065, start_date_:1748192400000000, last_date_:1748624400003705, this_date_:0, next_date_:1748710800000000, end_date_:64060560000000000, total_:347108, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:4, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748181773502269, start_date_:1748181600000000, last_date_:1748181600002317, this_date_:0, next_date_:1748786400000000, end_date_:64060560000000000, total_:173499952, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748527281005724, start_date_:1748527200000000, last_date_:1748527200011176, this_date_:0, next_date_:1749132000000000, end_date_:64060560000000000, total_:80994548, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748437312134789, start_date_:1748354400000000, last_date_:0, this_date_:0, next_date_:1748959200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"", func_type_:2}, {tenant_id_:1, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1748440876223997, start_date_:1748440800000000, last_date_:1748440800003333, this_date_:0, next_date_:1749045600000000, end_date_:64060560000000000, total_:76220664, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:0, max_failures_:0, state_:"SCHEDULED", func_type_:2}]) [2025-05-31 02:59:34.557487] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:504) [83020][T1_DBMSSched][T1][Y0-0000000000000000-0-0] [lt=152] check all tenants(ret=0) [2025-05-31 02:59:34.561200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.564658] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=48] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:34.564727] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=2, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:34.566224] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:34.572280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.582578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.582670] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:34.584283] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [33020][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=120] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:34.592882] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.592948] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.592979] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.593015] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.593041] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.593065] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.593113] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:34.598580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.608652] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:34.609217] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:34.609275] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:34.609527] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=163][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:34.609561] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:34.609995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=371][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.617210] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=26638) [2025-05-31 02:59:34.622186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.623348] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81740][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:34.632393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.633612] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=52] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:34.643225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.651473] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:34.654101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.658282] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F4-0-0] [lt=34][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525423840451, cache_obj->added_lc()=false, cache_obj->get_object_id()=53730, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:34.661212] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F6-0-0] [lt=48][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:34.661909] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F7-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750568914413452148e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544337160538230364e+03) [2025-05-31 02:59:34.662475] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F8-0-0] [lt=93][errcode=-4201] hashtable not init [2025-05-31 02:59:34.662685] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=14] get number(ret=0, service_type_=0, range=1, base_id=1748678374659052000, start_id=1748678374659052000, end_id=1748678374659052001) [2025-05-31 02:59:34.664679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.673083] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=68] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4323, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:34.673194] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=107][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137456731}], scn_array=[cnt:1, {val:1748678374659052000, v:0}], curr_size=566, accum_size=1277, time_cost=10028) [2025-05-31 02:59:34.673236] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1269, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:34.673329] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3561) [2025-05-31 02:59:34.673434] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:34.674315] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12B8-0-0] [lt=30] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:34.674398] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12B8-0-0] [lt=81] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6206, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:34.675623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=473][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.679549] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678374163506002, v:0}, new_max_version={val:1748678374163506002, v:0}, last_min_version={val:1748678374163506000, v:0}, last_max_version={val:1748678374163506000, v:0}, total_time_us=13142, delta_us=516039) [2025-05-31 02:59:34.679592] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678374163506002, v:0}, min_version={val:1748678374163506002, v:0}, max_version={val:1748678374163506002, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:34.683710] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38FA-0-0] [lt=70][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525423865878, cache_obj->added_lc()=false, cache_obj->get_object_id()=53731, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:34.685967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.686761] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:34.688835] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC10870A-0006366715B99E3F-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1748678374687994000, start_id=1748678374687994000, end_id=1748678374687994001) [2025-05-31 02:59:34.696969] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.707327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.717961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.720425] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F6E404D-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1748678374716026000, start_id=1748678374716026000, end_id=1748678374716026001) [2025-05-31 02:59:34.728331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.738653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.747351] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=55] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:34.749189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.758089] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20E1-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:34.760420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.766541] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678374665510000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678374766479}, version_delta=101028) [2025-05-31 02:59:34.770985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.779949] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:34.779991] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-05-31 02:59:34.780003] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:34.780018] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:34.780171] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525422959121, flush_io_finished_ret_:0, flush_io_finished_round_:19942, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:34.783259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.793708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.806410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.817072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.827339] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.838561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=137][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.848795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.851598] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:34.861258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=174][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.862653] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:34.862691] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=27] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53228740, tenant_id=1) [2025-05-31 02:59:34.862716] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:34.871718] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.885506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.886139] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=30] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19330215936, fast_delete_log_mode=false) [2025-05-31 02:59:34.887267] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:34.896247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.906709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.917213] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.927616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.939268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.949534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.957123] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A7-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:34.960000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=143][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.970270] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.971364] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=101] success to update min reserved snapshot(reserved_snapshot=1748676572256477000, duration=1800, snapshot_gc_ts=1748678372256477000) [2025-05-31 02:59:34.974744] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [81743][T1_TimerWK2_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=416] update reserved snapshot finished(cost_ts=3384, reserved_snapshot=1748676572256477000) [2025-05-31 02:59:34.977621] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=47] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:34.980496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:34.991212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.001562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.012259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.022653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.032926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.043199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.051894] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:35.053330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.064137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.065283] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=61] ====== check clog disk timer task ====== [2025-05-31 02:59:35.065313] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81740][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:35.071395] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:60K/13:493241 disk:60K/13:493241 t1m2:60K/13:493241 t1m2g0:60K/13:493241 |q0: 0 |q1: 0 [2025-05-31 02:59:35.076449] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.089051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.089558] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:35.091355] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=29] current monitor number(seq_=-1) [2025-05-31 02:59:35.093470] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=26] gc stale ls task succ [2025-05-31 02:59:35.099286] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.108999] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=2079744) [2025-05-31 02:59:35.109582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.118148] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:35.119943] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.121317] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=32] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:35.122701] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81744][T1_TimerWK3_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=37] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:35.131115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.133928] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:35.141341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.151485] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.162776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.167866] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678375167861, bytes: 7140437780, bw: 0.172563 MB/s, add_ts: 1004930, add_bytes: 181837 [2025-05-31 02:59:35.167939] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678374866981000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=300948, in_cluster_service=true, cluster_version={val:1748678374829151000, v:0}, min_cluster_version={val:1748678374829151000, v:0}, max_cluster_version={val:1748678374829151000, v:0}, get_cluster_version_err=0, cluster_version_delta=338778, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195590, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=366113171, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:35.169031] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [32846][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA7-0-0] [lt=11] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678375205193, keepalive_ts:-1}) [2025-05-31 02:59:35.169079] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [32846][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA7-0-0] [lt=45] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678375205193, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:35.175108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.185629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.198499] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.208817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.212938] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1748678375212933, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007672, add_bytes: 0 [2025-05-31 02:59:35.220604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.230910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.241569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.252087] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.254979] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:35.262238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.272795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.283242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.293561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.295779] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:35.304235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.314706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.325354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.335781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.346217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.357238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.358217] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.359753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.359866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=69] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.359903] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.367929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.378266] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.381576] INFO pn_ratelimit (group.c:653) [81064][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=66] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:35.386250] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=72, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:35.386436] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=151] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=14, avg_log_batch_cnt=1, total_group_log_size=6178, avg_group_log_size=441) [2025-05-31 02:59:35.386462] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=6178) [2025-05-31 02:59:35.386478] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:35.388019] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=6300) [2025-05-31 02:59:35.391670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [33405][T1_L0_G1][T1][YB42AC10871E-0006366549CC12C4-0-0] [lt=49] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=14, avg_log_gen_to_freeze_time=279, avg_log_gen_to_submit_time=393, avg_log_submit_to_flush_time=2911, avg_log_submit_to_slide_time=4171) [2025-05-31 02:59:35.391934] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.395255] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=6300, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:35.402599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.413236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=74][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.423479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.435170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.447976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.456455] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:35.458919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.469221] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.478223] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [34633][T1_L0_G1][T1][YB42AC10871E-0006366549CC12C5-0-0] [lt=26] init point thread id with(&point=0x55c0908e4600, idx_=1865, point=[thread id=34633, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=34633) [2025-05-31 02:59:35.480381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.490980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.493095] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:35.493154] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4388, iops:14, norm_iops:3, bw:60950, [delay/us]:prepare:16, schedule:318, submit:498, rt:1609, total:2443) [2025-05-31 02:59:35.493173] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:35.493232] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4389, avg_iops:14, avg_bw:60950, [delay/us]: prepare:16, schedule:318, submit:498, device:1609, total:2443) [2025-05-31 02:59:35.493246] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:35.493272] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4063, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:577, total:590) [2025-05-31 02:59:35.493297] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:35.493326] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4063, [delay/us]: prepare:0, schedule:0, submit:600000000, device:577, total:590) [2025-05-31 02:59:35.493344] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:35.493392] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:35.493401] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:35.493410] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:35.493418] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:35.493427] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=179kB/s, net_out=160kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:35.493446] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399470][2nd=2658]}) [2025-05-31 02:59:35.497367] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:35.498805] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2025-05-31 02:59:35.501189] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.511442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.517190] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA95-0-0] [lt=88][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:35.521686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.532072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.542694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.555702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.560801] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] ====== checkpoint timer task ====== [2025-05-31 02:59:35.561521] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=37] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:35.561586] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=64] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:35.562159] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:35.562369] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=203] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:35.562445] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=68] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:35.562466] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=0, rec_log_ts={val:1748678373736140001, v:0}) [2025-05-31 02:59:35.562495] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:35.562507] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:35.562521] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:35.562541] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:35.562579] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:35.562603] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:35.562625] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:35.562657] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81740][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=29] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:35.566550] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.566855] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=676, get_gts_with_stc_cnt=36, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=35, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:35.566969] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=111] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678375566817], gts:1748678375563922000, latest_srr:[mts=1748678375566817]}) [2025-05-31 02:59:35.566993] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:35.577540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.588033] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.588147] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:35.588363] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [33875][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=62] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:35.591460] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.591516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.591551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.591577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.591602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.591667] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.592676] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:35.601012] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83080][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA93-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525424783180, cache_obj->added_lc()=false, cache_obj->get_object_id()=53732, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:35.601460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.610754] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-05-31 02:59:35.612236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.623590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.634407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.634755] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:35.646508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.657108] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38FC-0-0] [lt=50][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525424839277, cache_obj->added_lc()=false, cache_obj->get_object_id()=53733, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:35.657294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.658074] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:35.669707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.675289] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFC07-0-0] [lt=55] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750568914413452148e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544337160538230364e+03) [2025-05-31 02:59:35.682124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.682509] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=2188, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:35.682548] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=232, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:35.682657] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=6178) [2025-05-31 02:59:35.682685] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=28, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:35.683993] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12C6-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=11, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:35.684014] INFO [LIB] stat (utility.h:1211) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12C6-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=4341, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:35.687002] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=538] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678375271195001, v:0}, new_max_version={val:1748678375271195001, v:0}, last_min_version={val:1748678375066939001, v:0}, last_max_version={val:1748678375066939001, v:0}, total_time_us=14347, delta_us=415803) [2025-05-31 02:59:35.687056] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678375271195001, v:0}, min_version={val:1748678375271195001, v:0}, max_version={val:1748678375271195001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:35.687082] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:35.695331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.695944] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A38FE-0-0] [lt=143][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525424878112, cache_obj->added_lc()=false, cache_obj->get_object_id()=53734, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:35.696344] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC10870A-0006366715B99E41-0-0] [lt=33] get number(ret=0, service_type_=0, range=1, base_id=1748678375689346000, start_id=1748678375689346000, end_id=1748678375689346001) [2025-05-31 02:59:35.700258] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:35.706231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.716993] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.729128] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.742191] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D514E-0-0] [lt=47][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525424924359, cache_obj->added_lc()=false, cache_obj->get_object_id()=53735, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:35.742578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.753148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.765179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.776156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.780393] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=50] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:35.780459] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=57] temporary file module use no blocks [2025-05-31 02:59:35.780473] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:35.780488] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:35.780503] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525423962407, flush_io_finished_ret_:0, flush_io_finished_round_:19943, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:35.788766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.799177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.810140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.823051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.833598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.846824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.857433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.860403] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:35.867686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.874281] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:164) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=601] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1, min_version={val:1748678375672624000, v:0}, base_version={val:1748678375271195001, v:0}, server_count=6, svr_array=[{addr:"172.16.135.166:2882", version:{val:1748678375672624000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678375874255, is_skipped:false}, {addr:"172.16.135.30:2882", version:{val:1748678375672624000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678375907539, is_skipped:false}, {addr:"172.16.135.167:2882", version:{val:1748678375672624000, v:0}, valid_part_count:1, total_part_count:1, generate_tstamp:1748678375839518, is_skipped:false}, {addr:"172.16.135.20:2882", version:{val:1748678370852785000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678375811947, is_skipped:true}, {addr:"172.16.135.10:2882", version:{val:1748678370691489000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678375806914, is_skipped:true}, {addr:"172.16.135.168:2882", version:{val:1748678370837422000, v:0}, valid_part_count:0, total_part_count:0, generate_tstamp:1748678375808602, is_skipped:true}], skip_server_count=3, skip_servers=["172.16.135.20:2882", "172.16.135.10:2882", "172.16.135.168:2882"]) [2025-05-31 02:59:35.878462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.889272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.901019] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:35.901740] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.912479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.924567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=170][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.936699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.948136] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.961147] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A8-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:35.961630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.971958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.977925] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=43] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:35.980933] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [83008][T1_ArcSender][T1][YB42AC1087A6-000636654289FC95-0-0] [lt=40][errcode=-4018] try free send task failed(ret=-4018) [2025-05-31 02:59:35.982758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:35.984304] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [83006][T1_ArcFetcher][T1][YB42AC1087A6-000636654279FC95-0-0] [lt=31] ObArchiveFetcher is running(thread_index=0) [2025-05-31 02:59:35.993410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.004981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.015689] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.026465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.036803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.047273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.057479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.063288] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:36.068447] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=8, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:36.073194] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:990) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-05-31 02:59:36.073355] INFO [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:660) [81753][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=31] tenant does not have duplicate ls(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1) [2025-05-31 02:59:36.078949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.079631] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:44K/10:225538 disk:44K/10:225538 t1m2:44K/10:225538 t1m2g0:44K/10:225538 |q0: 0 |q1: 0 [2025-05-31 02:59:36.089935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.093680] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=39] gc stale ls task succ [2025-05-31 02:59:36.094099] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137464330}], scn_array=[cnt:1, {val:1748678376074669000, v:0}], curr_size=589, accum_size=2599, time_cost=14682) [2025-05-31 02:59:36.105022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.105641] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:36.115802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.115879] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=181] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-05-31 02:59:36.118367] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:36.122720] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:36.126606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.135105] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:36.137100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.147327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.159686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.170260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.174145] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1748678376174141, bytes: 7140579805, bw: 0.134600 MB/s, add_ts: 1006280, add_bytes: 142025 [2025-05-31 02:59:36.181067] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=33] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-31 02:59:36.181171] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=90] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-31 02:59:36.181190] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [81740][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=961) [2025-05-31 02:59:36.182964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.193285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.194224] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] tx gc loop thread is running(MTL_ID()=1) [2025-05-31 02:59:36.194257] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=31] try gc retain ctx [2025-05-31 02:59:36.194266] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-31 02:59:36.194312] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1748678376074669000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1748678370930413000, v:0}, keep_alive_scn_:{val:1748678371137961001, v:0}, update_ts_:1748678376193452}) [2025-05-31 02:59:36.195266] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-31 02:59:36.203541] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.214292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.222421] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1748678376222416, bytes: 0, bw: 0.000000 MB/s, add_ts: 1009483, add_bytes: 0 [2025-05-31 02:59:36.224543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.234810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.246732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.256981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.264779] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=566208, replace_num=15728) [2025-05-31 02:59:36.264815] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:36.268723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=160][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.279165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.290170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.302805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.307739] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:36.313185] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.323414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.333669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.343994] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.349197] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207D-0-0] [lt=30] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:36.349341] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207D-0-0] [lt=90] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=103) [2025-05-31 02:59:36.354381] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.358744] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.358938] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.359915] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.360641] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.364596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.375245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.388255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.394958] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=115] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=80, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:36.395076] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3423, avg_group_log_size=380) [2025-05-31 02:59:36.395114] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3423) [2025-05-31 02:59:36.395180] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:36.396373] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=80] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:36.396423] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=48] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:36.396511] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109C-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:36.396554] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81451][SerScheQueue1][T0][YB42AC1087A6-0006366546FA109C-0-0] [lt=39] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:36.398346] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3423) [2025-05-31 02:59:36.401039] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=15, avg_log_gen_to_submit_time=121, avg_log_submit_to_flush_time=6907, avg_log_submit_to_slide_time=7411) [2025-05-31 02:59:36.401515] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=48] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3423, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:36.401751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:36.402188] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [35611][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:36.402590] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=196] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:36.413109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.425065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=114][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.435415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.446926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.457194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.465429] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:36.467479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.478798] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.489130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.499661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.505733] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:36.505792] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, norm_iops:2, bw:40448, [delay/us]:prepare:16, schedule:261, submit:130, rt:4916, total:5324) [2025-05-31 02:59:36.505811] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:36.505977] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40444, [delay/us]: prepare:16, schedule:261, submit:130, device:4916, total:5324) [2025-05-31 02:59:36.506000] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:36.506031] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4044, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:284, total:300) [2025-05-31 02:59:36.506117] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:36.506157] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4044, [delay/us]: prepare:0, schedule:0, submit:600000000, device:284, total:300) [2025-05-31 02:59:36.506176] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:36.506193] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:36.506202] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:36.506210] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:36.506218] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:36.506227] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=123kB/s, net_out=96kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:36.506249] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399471][2nd=2940]}) [2025-05-31 02:59:36.509263] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-54898688, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:36.510000] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.514346] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=29] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:36.516309] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=29] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:36.516411] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81744][T1_TimerWK3_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678372256477000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:36.520482] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81742][T1_TimerWK1_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:36.520560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.530923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.538222] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83085][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA96-0-0] [lt=69][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:36.542250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.552896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.563187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.568919] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34] ts response task statistics(alloc_count=487178, free_count=243588) [2025-05-31 02:59:36.568971] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:36.569030] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678376068121], gts:1748678376106914000, latest_srr:[mts=1748678376568991]}) [2025-05-31 02:59:36.574035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.584356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.591006] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.592803] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.592927] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.593028] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.593118] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.593154] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.593247] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:36.596793] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.600129] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A79-0-0] [lt=32] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:36.607125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.610501] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:242) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=31] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"172.16.135.166:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}]) [2025-05-31 02:59:36.615160] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=48] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-05-31 02:59:36.617427] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:192) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=31] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-05-31 02:59:36.618280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.619017] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=2079744) [2025-05-31 02:59:36.619261] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:230) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=20] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-05-31 02:59:36.619280] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:152) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=18] refresh resource mapping rule success(tenant_id=1, plan=) [2025-05-31 02:59:36.621127] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:147) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=8] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-05-31 02:59:36.621195] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:229) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=20] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-05-31 02:59:36.621209] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:119) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=12] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-05-31 02:59:36.621234] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:515) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=14] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1003->{value:""}, 1004->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} }) [2025-05-31 02:59:36.621737] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=499] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:36.621778] INFO set_nway (ob_tenant_mutil_allocator.cpp:387) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=32] finish set nway(tenant_id_=1, nway=3) [2025-05-31 02:59:36.621789] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:398) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-05-31 02:59:36.621810] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=19] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2b0904d78170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-05-31 02:59:36.621831] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:407) [81625][OmtNodeBalancer][T1][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=20] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-05-31 02:59:36.625773] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=6] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:36.625806] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=31] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1003, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:36.625816] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:217) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=8] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1004, value_updated=true, data_version={version_str:"4.3.5.1", version_val:17180067073}) [2025-05-31 02:59:36.625927] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=74] cal_all_part_disk_default_percentage succ(data_dir="/data/ob/data/sstable", clog_dir="/data/ob/data/clog", shared_mode=true, data_disk_total_size=110716850176, data_disk_default_percentage=60, clog_disk_total_size=110716850176, clog_disk_default_percentage=30) [2025-05-31 02:59:36.625942] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=14] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=110716850176, disk_size=15032385536) [2025-05-31 02:59:36.625951] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=110716850176) [2025-05-31 02:59:36.625964] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=9] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=110716850176, this={dir::"/data/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:114, max_block_id:313, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2025-05-31 02:59:36.625981] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [81625][OmtNodeBalancer][T0][YB42AC1087A6-0006366541CA0BE2-0-0] [lt=16] refresh tenant config(tenants=[1, 1003, 1004], ret=0) [2025-05-31 02:59:36.628532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.630535] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:36.635945] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:36.639046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.650900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.661512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.666060] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:36.671750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.682009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.682037] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFC16-0-0] [lt=72] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.750568914413452148e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=287, Throttle Trigger(MB)=172, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.544337160538230364e+03) [2025-05-31 02:59:36.685028] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [81740][T1_TimerWK0_Flush][T1][Y0-0000000000000000-0-0] [lt=12] ====== traversal_flush timer task ====== [2025-05-31 02:59:36.685390] INFO [MDS] flush (mds_table_mgr.cpp:184) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F56-0-0] [lt=36] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1748671209934467003, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1748671210167354001, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1748671210376323001, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1748671210376323002, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1748671210543332001, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1748671210543332002, v:0}}, {tablet_id:{id:60429}, rec_scn:{val:1748671210670946001, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1748671210670946002, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1748671210670946005, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1748671210670946006, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1748671211024177001, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1748671211024177002, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1748671211024177004, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1748671211024177006, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1748671211024177007, v:0}}, {tablet_id:{id:1152921504606846994}, rec_scn:{val:1748671211428913005, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1748671211927662001, v:0}}, {tablet_id:{id:50124}, rec_scn:{val:1748671211927662002, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1748671211927662004, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1748671211927662005, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1748671211927662007, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1748671211927662009, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1748671211927662011, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1748671211927662012, v:0}}, {tablet_id:{id:109}, rec_scn:{val:1748671212522642001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1748671212522642002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1748671212635825001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1748671212635825002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1748671212719809001, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1748671212719809002, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1748671212719809003, v:0}}, }, max_consequent_callbacked_scn={val:1748678376475346000, v:0}, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:36.685545] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [81740][T1_TimerWK0_Flush][T1][YB42AC1087A6-00063665444A5F56-0-0] [lt=147] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-31 02:59:36.687186] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12CF-0-0] [lt=8] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3423) [2025-05-31 02:59:36.690024] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5563, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:36.690075] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=941, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:36.690129] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=13, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:36.690154] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=7802, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:36.690173] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=60, this=0x2b09131effe0, extra_info="") [2025-05-31 03:59:36.690319] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [35611][T1_L0_G1][T1][YB42AC1087A7-00063665517F5709-0-0] [lt=59] init point thread id with(&point=0x55c0908f3a80, idx_=2843, point=[thread id=35611, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=35611) [2025-05-31 02:59:36.690764] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678376475346000, v:0}, new_max_version={val:1748678376475346000, v:0}, last_min_version={val:1748678376275820000, v:0}, last_max_version={val:1748678376275820000, v:0}, total_time_us=10478, delta_us=215415) [2025-05-31 02:59:36.690816] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678376475346000, v:0}, min_version={val:1748678376475346000, v:0}, max_version={val:1748678376475346000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:36.694031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.704964] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3900-0-0] [lt=67][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525425887133, cache_obj->added_lc()=false, cache_obj->get_object_id()=53736, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:36.705620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.709949] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:36.716518] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.726739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.737003] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.750562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.760782] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.765930] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20E2-0-0] [lt=16] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:36.771355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.781012] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=56] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:36.781108] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=85] temporary file module use no blocks [2025-05-31 02:59:36.781121] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:36.781138] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:36.781154] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525424962746, flush_io_finished_ret_:0, flush_io_finished_round_:19944, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:36.781746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.789787] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-31 02:59:36.793040] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:131) [81740][T1_TimerWK0_ReqMemEvict][T1][YB42AC1087A6-00063665444A5F57-0-0] [lt=36] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=33) [2025-05-31 02:59:36.793283] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] get number(ret=0, service_type_=0, range=1, base_id=1748678376793231000, start_id=1748678376793231000, end_id=1748678376793231001) [2025-05-31 02:59:36.794065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.805431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.806305] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=27] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-31 02:59:36.806355] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=43] start lib cache evict(tenant_id=1, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:36.806372] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=48234496, mem_limit=53687090, cache_obj_num=314, cache_node_num=313) [2025-05-31 02:59:36.806383] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:36.809139] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=314, alloc_obj_list=[{obj_id:31, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66482}, {obj_id:551, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64939}, {obj_id:52044, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74789}, {obj_id:52046, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70896}, {obj_id:52047, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48841}, {obj_id:52049, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47716}, {obj_id:52050, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47675}, {obj_id:52051, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52054, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50170}, {obj_id:52055, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57722}, {obj_id:52056, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59423}, {obj_id:52057, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57025}, {obj_id:52058, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48740}, {obj_id:52060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54964}, {obj_id:52061, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73037}, {obj_id:52062, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53651}, {obj_id:52063, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47104}, {obj_id:52065, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50097}, {obj_id:52066, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120728}, {obj_id:52069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50991}, {obj_id:52070, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65787}, {obj_id:52071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46712}, {obj_id:52072, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:44800}, {obj_id:52074, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64735}, {obj_id:52075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49180}, {obj_id:52077, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57572}, {obj_id:52093, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55769}, {obj_id:52098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47563}, {obj_id:52102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47944}, {obj_id:52111, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47336}, {obj_id:52112, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47611}, {obj_id:52113, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283782}, {obj_id:52115, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526095}, {obj_id:52116, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524437}, {obj_id:52117, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526110}, {obj_id:52118, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:107176}, {obj_id:52119, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526044}, {obj_id:52120, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525088}, {obj_id:52121, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526074}, {obj_id:52122, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52947}, {obj_id:52123, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525086}, {obj_id:52124, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525761}, {obj_id:52125, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:526562}, {obj_id:52126, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:524324}, {obj_id:52127, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525841}, {obj_id:52128, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525090}, {obj_id:52129, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525649}, {obj_id:52130, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:525820}, {obj_id:52131, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241039}, {obj_id:52132, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84229}, {obj_id:52133, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53798}, {obj_id:52134, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51850}, {obj_id:52212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47816}, {obj_id:52213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48552}, {obj_id:52217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60368}, {obj_id:52220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:83264}, {obj_id:52221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:79775}, {obj_id:52222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78207}, {obj_id:52223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47640}, {obj_id:52226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:61485}, {obj_id:52227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:73743}, {obj_id:52228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49100}, {obj_id:52230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:81718}, {obj_id:52231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51496}, {obj_id:52232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:67016}, {obj_id:52233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59467}, {obj_id:52234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157508}, {obj_id:52240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63459}, {obj_id:52241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47453}, {obj_id:52242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51021}, {obj_id:52243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63026}, {obj_id:52244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47441}, {obj_id:52245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47490}, {obj_id:52246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52041}, {obj_id:52247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47444}, {obj_id:52248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57587}, {obj_id:52249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49629}, {obj_id:52250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62690}, {obj_id:52251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46152}, {obj_id:52252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52253, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48163}, {obj_id:52255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54245}, {obj_id:52257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52193}, {obj_id:52258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:120608}, {obj_id:52260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46456}, {obj_id:52261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97645}, {obj_id:52263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:45232}, {obj_id:52267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60986}, {obj_id:52268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49416}, {obj_id:52269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:89824}, {obj_id:52270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60530}, {obj_id:52271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55134}, {obj_id:52272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50703}, {obj_id:52273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:63211}, {obj_id:52274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:59122}, {obj_id:52277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55344}, {obj_id:52278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50386}, {obj_id:52279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46056}, {obj_id:52280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62935}, {obj_id:52282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47535}, {obj_id:52284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47761}, {obj_id:52285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55183}, {obj_id:52286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:66321}, {obj_id:52287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70251}, {obj_id:52289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60906}, {obj_id:52290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64192}, {obj_id:52291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52063}, {obj_id:52294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52793}, {obj_id:52295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74202}, {obj_id:52298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:92710}, {obj_id:52299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54124}, {obj_id:52301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62067}, {obj_id:52302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47487}, {obj_id:52303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52857}, {obj_id:52304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:57871}, {obj_id:52305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:96085}, {obj_id:52306, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58016}, {obj_id:52307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53425}, {obj_id:52311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72140}, {obj_id:52313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72223}, {obj_id:52314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71241}, {obj_id:52315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47024}, {obj_id:52316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47762}, {obj_id:52332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:86245}, {obj_id:52333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52240}, {obj_id:52334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50624}, {obj_id:52335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47920}, {obj_id:52336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76069}, {obj_id:52337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47083}, {obj_id:52338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54212}, {obj_id:52339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46192}, {obj_id:52340, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144957}, {obj_id:52341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55613}, {obj_id:52342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47055}, {obj_id:52344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52348, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:50567}, {obj_id:52350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49008}, {obj_id:52371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46952}, {obj_id:52374, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48520}, {obj_id:52376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52377, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64025}, {obj_id:52388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69324}, {obj_id:52390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46960}, {obj_id:52409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54319}, {obj_id:52413, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52427, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52428, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52429, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52442, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55179}, {obj_id:52443, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94031}, {obj_id:52451, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:76509}, {obj_id:52452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:54737}, {obj_id:52460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48544}, {obj_id:52475, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:74048}, {obj_id:52481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:94636}, {obj_id:52482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:53476}, {obj_id:52496, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252328}, {obj_id:52498, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105453}, {obj_id:52535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46563}, {obj_id:52536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47096}, {obj_id:52537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47112}, {obj_id:52538, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47520}, {obj_id:52578, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93417}, {obj_id:52579, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52623, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49942}, {obj_id:52686, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52687, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52737, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:91739}, {obj_id:52752, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:52939, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:121684}, {obj_id:52940, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:69538}, {obj_id:53069, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178951}, {obj_id:53091, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53092, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72226}, {obj_id:53094, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62932}, {obj_id:53096, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53097, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53098, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53152, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53157, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53161, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53165, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93212}, {obj_id:53166, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53188, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53189, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53190, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53197, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56281}, {obj_id:53198, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53200, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53201, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53202, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53203, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:62983}, {obj_id:53219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:124385}, {obj_id:53220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:139398}, {obj_id:53221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127110}, {obj_id:53223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:144367}, {obj_id:53224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:163763}, {obj_id:53225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102648}, {obj_id:53226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175792}, {obj_id:53227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:127750}, {obj_id:53229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:93905}, {obj_id:53376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:102055}, {obj_id:53378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14753}, {obj_id:53380, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52909}, {obj_id:53381, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60232}, {obj_id:53382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53383, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1301275}, {obj_id:53384, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56585}, {obj_id:53385, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53387, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:959407}, {obj_id:53390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:135613}, {obj_id:53392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:111023}, {obj_id:53393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:85186}, {obj_id:53405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:78333}, {obj_id:53406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:110376}, {obj_id:53409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:133406}, {obj_id:53561, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53562, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53563, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53564, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:53694, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:10671, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49120}, {obj_id:11882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47672}, {obj_id:11887, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:49387}, {obj_id:21576, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65663}, {obj_id:24484, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:97648}, {obj_id:24486, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:60632}, {obj_id:24492, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55615}, {obj_id:32883, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70923}, {obj_id:33971, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:58630}, {obj_id:33973, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48522}, {obj_id:36991, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:80748}, {obj_id:36992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55051}, {obj_id:41732, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:41967, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:48968}, {obj_id:42068, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47679}, {obj_id:42071, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:56010}, {obj_id:42075, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:51274}, {obj_id:42085, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:77331}, {obj_id:42100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:55696}, {obj_id:43236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:71018}, {obj_id:43365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:84366}, {obj_id:44168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:65579}, {obj_id:46494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:70256}, {obj_id:46499, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72592}, {obj_id:46508, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52154}, {obj_id:46513, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:72617}, {obj_id:46523, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:52976}, {obj_id:46524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:47270}, {obj_id:46526, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:105781}, {obj_id:46537, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:64868}, {obj_id:46820, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:46308}]) [2025-05-31 02:59:36.809950] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [81742][T1_TimerWK1_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=805] schedule next cache evict task(evict_interval=5000000) [2025-05-31 02:59:36.818647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.828944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.839292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.849494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.859766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.866254] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:36.871627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.876581] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:36.876621] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=30] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:59:36.876646] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=15] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:36.880977] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:683) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] no need to update min/max version(new_version={val:1748678376475346000, v:0}, min_version={val:1748678376475346000, v:0}, max_version={val:1748678376475346000, v:0}) [2025-05-31 02:59:36.881080] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=96] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:36.882020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.892288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.904193] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.911731] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:36.914481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.924748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.935014] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.945819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.956386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.964181] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44A9-0-0] [lt=18][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:36.973214] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC10870A-000636671698E3FC-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525426155383, cache_obj->added_lc()=false, cache_obj->get_object_id()=53737, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:36.973795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.979209] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=403] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:36.985582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:36.996205] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.007183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.018065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.028399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=185][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.038777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.039988] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:705) [81064][TimerWK2_LocalityReload][T0][Y0-0000000000000000-0-0] [lt=32] runTimer to refresh locality_info(ret=0) [2025-05-31 02:59:37.045006] INFO [SHARE] set_server_locality_array (ob_server_locality_cache.cpp:345) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] set_server_locality_array success(server_locality_array=[{inited:true, addr:"172.16.135.10:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666222436391, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.166:2882", zone:"zone1", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658531155476, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.167:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658532263276, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.168:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666660298916, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.20:2882", zone:"zone2", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748666375829119, server_stop_time:0, server_status:1}, {inited:true, addr:"172.16.135.30:2882", zone:"zone3", zone_type:0, idc:"", region:"sys_region", is_active:true, start_service_time:1748658461872039, server_stop_time:0, server_status:1}], has_readonly_zone=false) [2025-05-31 02:59:37.045090] INFO [SHARE] load_region (ob_locality_table_operator.cpp:330) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=70] tenant_schema primary_zone is NULL, or no need calc region priority(tenant_id=1, primary_zone=RANDOM) [2025-05-31 02:59:37.045105] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=14] add_locality_zone success(locality_zone={tenant_id:1, region_priority:18446744073709551615}) [2025-05-31 02:59:37.045128] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=9] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}]}) [2025-05-31 02:59:37.045217] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=87] add_locality_zone success(locality_zone={tenant_id:1003, region_priority:0}) [2025-05-31 02:59:37.045245] INFO [SHARE] load_region (ob_locality_table_operator.cpp:340) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=11] get_primary_zone_prioriry(primary_zone=zone1;zone2,zone3, tenant_primary_region_array=[{region:"sys_region", region_priority:0, zone_array:["zone1", "zone2", "zone3"]}], locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}]}) [2025-05-31 02:59:37.045262] INFO [SHARE] load_region (ob_locality_table_operator.cpp:357) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16] add_locality_zone success(locality_zone={tenant_id:1004, region_priority:0}) [2025-05-31 02:59:37.045272] INFO [SHARE] load_region (ob_locality_table_operator.cpp:367) [81517][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=4] load region(ret=0, locality_info={version:0, local_region:"sys_region", local_zone:"zone1", local_idc:"", local_zone_type:0, local_zone_status:2, locality_region_array:[{region:"sys_region", region_priority:18446744073709551615, zone_array:["zone1", "zone2", "zone3"]}], locality_zone_array:[{tenant_id:1, region_priority:18446744073709551615}, {tenant_id:1003, region_priority:0}, {tenant_id:1004, region_priority:0}]}) [2025-05-31 02:59:37.049664] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.060669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=72][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.066455] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:37.068032] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-05-31 02:59:37.068064] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81742][T1_TimerWK1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:37.072139] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1748678377068890000, start_id=1748678377068890000, end_id=1748678377068890001) [2025-05-31 02:59:37.072448] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678376569326], gts:1748678376622741000, latest_srr:[mts=1748678377072426]}) [2025-05-31 02:59:37.072743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.083375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.094174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.095426] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=47] current monitor number(seq_=-1) [2025-05-31 02:59:37.096120] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=28] gc stale ls task succ [2025-05-31 02:59:37.106378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.112622] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=62] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:37.115288] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [32846][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD45-0-0] [lt=50] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678377114248, keepalive_ts:-1}) [2025-05-31 02:59:37.115383] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [32846][T1_L0_G1][T1][YB42AC1087A7-00063665463DBD45-0-0] [lt=92] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.167:2882", region:"sys_region", register_time_us:1748678377114248, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:37.116688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.118580] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:37.119351] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:37.123992] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:37.127203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.136119] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:37.138948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.149404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.161530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.171988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.175729] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1748678377175723, bytes: 7140738129, bw: 0.150751 MB/s, add_ts: 1001582, add_bytes: 158324 [2025-05-31 02:59:37.182434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.186141] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:36K/9:379677 disk:36K/9:379677 t1m2:36K/9:379677 t1m2g0:36K/9:379677 |q0: 0 |q1: 0 [2025-05-31 02:59:37.189488] WDIAG [SERVER] deliver_rpc_request (ob_srv_deliver.cpp:666) [81433][pnio1][T0][YB42AC10870A-000636671AB94822-0-0] [lt=13][errcode=-5150] can't deliver request(req={packet:{hdr_:{checksum_:4045938138, pcode_:1861, hlen_:192, priority_:5, flags_:6151, tenant_id_:1003, priv_tenant_id_:0, session_id_:0, trace_id_:12380982511370, timeout_:200000, timestamp:1748678377221185, dst_cluster_id:1748168748, cost_time:{len:40, arrival_push_diff:0, push_pop_diff:0, pop_process_start_diff:0, process_start_end_diff:0, process_end_response_diff:0, packet_id:1392523, request_arrival_time:0}, compressor_type_:0, original_len_:0, src_cluster_id_:1748168748, seq_no_:1748678377196020, data_version_:17180067073}, chid_:0, clen_:131, assemble:false, msg_count:0, payload:0}, type:0, group:0, sql_req_level:0, connection_phase:0, recv_timestamp_:1748678377189483, enqueue_timestamp_:0, request_arrival_time_:1748678377189483, trace_id_:Y0-0000000000000000-0-0}, ret=-5150) [2025-05-31 02:59:37.189573] WDIAG [SERVER] deliver (ob_srv_deliver.cpp:865) [81433][pnio1][T0][YB42AC10870A-000636671AB94822-0-0] [lt=68][errcode=-5150] deliver rpc request fail(&req=0x2b09b7e04098, ret=-5150) [2025-05-31 02:59:37.192529] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B0B-0-0] [lt=36][errcode=-4006] not init(ret=-4006) [2025-05-31 02:59:37.192565] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B0B-0-0] [lt=28] finish scan map holding items(scan_cnt=0) [2025-05-31 02:59:37.195686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.198557] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B0C-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:37.206421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=68][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.217147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.229265] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=49] PNIO [ratelimit] time: 1748678377229259, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006843, add_bytes: 0 [2025-05-31 02:59:37.229661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.239984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.253533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=250][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.264747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.266689] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:37.278311] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=277][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.288961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.299167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.311989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.313663] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:37.322602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.332784] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.343146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.353383] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.358460] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.358925] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.358967] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.358995] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.363733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.374139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.383567] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get number(ret=0, service_type_=0, range=1, base_id=1748678377378499000, start_id=1748678377378499000, end_id=1748678377378499001) [2025-05-31 02:59:37.384602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.395411] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.409815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=111][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.409926] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:37.410034] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [34598][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=51] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:37.422020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.432343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.442748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.454082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.464762] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.470656] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:37.475163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.485432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.495743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.496524] INFO [LIB] stat (utility.h:1211) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A0-0-0] [lt=84] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=84, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:37.496588] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A0-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=4927, avg_group_log_size=547) [2025-05-31 02:59:37.496612] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A0-0-0] [lt=20] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3859) [2025-05-31 02:59:37.496627] INFO [LIB] stat (utility.h:1211) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A0-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:37.497615] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=4927) [2025-05-31 02:59:37.500566] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12D6-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=113, avg_log_submit_to_flush_time=3618, avg_log_submit_to_slide_time=4995) [2025-05-31 02:59:37.500655] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4927, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:37.502110] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678377501819000, start_id=1748678377501819000, end_id=1748678377501819001) [2025-05-31 02:59:37.506021] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC1087A8-0006366730AE33E6-0-0] [lt=89][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525426688190, cache_obj->added_lc()=false, cache_obj->get_object_id()=53738, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:37.506230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.510223] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:37.510405] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4608, iops:8, norm_iops:2, bw:36695, [delay/us]:prepare:25, schedule:327, submit:29, rt:1992, total:2375) [2025-05-31 02:59:37.510429] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:37.510627] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=179] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4608, avg_iops:8, avg_bw:36693, [delay/us]: prepare:25, schedule:327, submit:29, device:1992, total:2375) [2025-05-31 02:59:37.510643] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:37.510672] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4077, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:453, total:467) [2025-05-31 02:59:37.510699] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:37.510811] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=106] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4077, [delay/us]: prepare:0, schedule:0, submit:600000000, device:453, total:467) [2025-05-31 02:59:37.510833] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:37.510915] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:37.510928] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:37.510937] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:37.510946] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:37.510955] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=174kB/s, net_out=101kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:37.510975] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399472][2nd=1671]}) [2025-05-31 02:59:37.514709] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:37.515371] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=8][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.515456] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=80][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.515477] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: B61085E969360600, int: 1748678377541814}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: B61085E969360600, int: 1748678377541814}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.515732] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:45) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=117] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=60708, insert_dup_release_count=60708, col_map_alloc_count=0, col_map_release_count=0) [2025-05-31 02:59:37.515814] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=62][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.516746] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2025-05-31 02:59:37.516889] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.519079] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=56] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:13006}, to_seq_no={branch:0, seq:9064}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382273} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.519217] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:674) [83059][T1_L0_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=77] rpc statistics(total_trans_msg_count=18, total_batch_msg_count=0) [2025-05-31 02:59:37.519232] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:148) [83059][T1_L0_G0][T1][YB42AC108714-000636671F1D63A1-0-0] [lt=13] transaction rpc statistics(total_rt=5294, total_process=29, avg_rt=176) [2025-05-31 02:59:37.528578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.530688] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63A2-0-0] [lt=44] get number(ret=0, service_type_=0, range=1, base_id=1748678377528620000, start_id=1748678377528620000, end_id=1748678377528620001) [2025-05-31 02:59:37.531036] INFO [STORAGE.TRANS] process (ob_trans_rpc.cpp:145) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63A2-0-0] [lt=42] transaction rpc statistics(total_rt=1743, total_process=7, avg_rt=217) [2025-05-31 02:59:37.539116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.540343] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A3-0-0] [lt=10][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.540400] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A3-0-0] [lt=51][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.540421] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A3-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f3, hex: AD8485E969360600, int: 1748678377571501}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4fb, hex: AD8485E969360600, int: 1748678377571501}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.540672] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A3-0-0] [lt=149][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.541928] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63A3-0-0] [lt=42] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7784}, to_seq_no={branch:0, seq:5095}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382274} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.551431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.554444] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83061][T1_L0_G0][T1][YB42AC108714-000636671F1D63A4-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1748678377554351000, start_id=1748678377554351000, end_id=1748678377554351001) [2025-05-31 02:59:37.563080] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA97-0-0] [lt=43][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:37.563880] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.564958] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83074][T1_L0_G0][T1][YB42AC108714-000636671F1D63A5-0-0] [lt=82][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.564992] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83074][T1_L0_G0][T1][YB42AC108714-000636671F1D63A5-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.565010] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83074][T1_L0_G0][T1][YB42AC108714-000636671F1D63A5-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b099940c820, local_datums:0x2b099940c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b099940c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b099940c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c4f0, hex: F5DB85E969360600, int: 1748678377593845}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c4f8, hex: F5DB85E969360600, int: 1748678377593845}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b099940c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.565129] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83074][T1_L0_G0][T1][YB42AC108714-000636671F1D63A5-0-0] [lt=65][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.569995] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63A5-0-0] [lt=46] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:12062}, to_seq_no={branch:0, seq:3674}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382275} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.574654] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:37.575694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.587167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.591211] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.596766] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38174][T1_L0_G5][T1][YB42AC1087A7-0006366541CDB896-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1748678377596202000, start_id=1748678377596202000, end_id=1748678377596202001) [2025-05-31 02:59:37.598413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.599522] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.599622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.599657] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.599683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.599708] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.599742] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:37.600936] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137472351}], scn_array=[cnt:1, {val:1748678377581947001, v:0}], curr_size=588, accum_size=2269, time_cost=10943) [2025-05-31 02:59:37.605782] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:395) [81065][TimerWK3_ServerTracerTim][T0][YB42AC1087A6-0006366545EA56F7-0-0] [lt=42] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-05-31 02:59:37.606313] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63A7-0-0] [lt=108][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.606369] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63A7-0-0] [lt=50][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.606391] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63A7-0-0] [lt=13][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b095c40c820, local_datums:0x2b095c40c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b095c40c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b095c40c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4f0, hex: 445C86E969360600, int: 1748678377626692}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4f8, hex: 445C86E969360600, int: 1748678377626692}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.606532] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63A7-0-0] [lt=66][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.608832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.611629] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63A7-0-0] [lt=46] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:21080}, to_seq_no={branch:0, seq:14889}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382276} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.612289] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:37.612322] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:37.612357] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:37.612379] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:37.619653] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:37.619940] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] start to print blacklist info [2025-05-31 02:59:37.620364] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=47] ls blacklist refresh finish(cost_time=29591) [2025-05-31 02:59:37.621065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=103][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.635681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.635945] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A9-0-0] [lt=39][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.635976] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A9-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.635995] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A9-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: 09FB86E969360600, int: 1748678377667337}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: 09FB86E969360600, int: 1748678377667337}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.636180] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:37.636216] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63A9-0-0] [lt=87][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.639503] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC108714-000636671F1D63A9-0-0] [lt=5] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6392}, to_seq_no={branch:0, seq:2640}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382277} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.648511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.659425] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.668663] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AB-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.668704] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AB-0-0] [lt=37][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.668722] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AB-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: 187E87E969360600, int: 1748678377700888}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: 187E87E969360600, int: 1748678377700888}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 03085B0800000000, int: 140183555}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 00A0B20900000000, int: 162701312}]}) [2025-05-31 02:59:37.668850] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AB-0-0] [lt=65][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.669667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.671017] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63AB-0-0] [lt=58] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:6571}, to_seq_no={branch:0, seq:2879}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382278} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.672309] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:37.682243] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.689730] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83059][T1_L0_G0][T1][YB42AC108714-000636671F1D63AC-0-0] [lt=9] get number(ret=0, service_type_=0, range=1, base_id=1748678377688607000, start_id=1748678377688607000, end_id=1748678377688607001) [2025-05-31 02:59:37.692182] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [32846][T1_L0_G1][T1][YB42AC10871E-0006366549CC12DD-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=7723) [2025-05-31 02:59:37.695682] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT WRITE LOG COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=3258, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:37.695720] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=430, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:37.695768] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=14, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:37.695788] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=5216, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:37.695808] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=14, stat_interval=1000000, avg (count/cost)=31, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:37.695883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=122][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.702262] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=12][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af3f50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382279} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5237} write_epoch=1, prev=this=0x2b0980339510 trans_version={val:1748677891786141000, v:0} scn={val:1748677891786141000, v:0} tx_id={txid:11381612} prev=0x2b0988a86c48 next=(nil) modify_count=11 acc_checksum=228536282 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:33302} write_epoch=0, snapshot_version={val:1748678377699808000, v:0}, *this={this=0x2b09b31611a8 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980339510 latest_compact_node=0x2b0980339510 max_trans_version={val:1748678317065939000, v:0} max_trans_id=11382200 max_elr_trans_version={val:1748678317065939000, v:0} max_elr_trans_id=11382200 latest_compact_ts=1748677891800321 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677891786141000, v:0} min_modify_scn={val:1748675827021622000, v:0}}) [2025-05-31 02:59:37.702326] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=62][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af3f50 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382279} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:5237} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407598432, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407598432, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678377699808000, v:0}, tx_id:{txid:11382279}, scn:{branch:0, seq:5234}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382279}, tx_desc:{this:0x2b094e99d560, tx_id:{txid:11382279}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678377700887, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678377733085, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:5237}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:37.702470] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=142][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382279}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:5237}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:37.702489] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=18][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382279} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:37.702520] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.702535] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: DEFB87E969360600, int: 1748678377733086}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: DEFB87E969360600, int: 1748678377733086}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: ACDB750000000000, int: 7723948}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 0090B40000000000, int: 11833344}]}) [2025-05-31 02:59:37.702645] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.706397] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=359][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.709107] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83099][T1_L0_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=51] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:11900}, to_seq_no={branch:0, seq:5235}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382279} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.715278] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83099][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3902-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525426897447, cache_obj->added_lc()=false, cache_obj->get_object_id()=53739, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:37.717420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.718122] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:37.724335] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63AD-0-0] [lt=11] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.702568960189819336e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=285, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.630333230382834699e+03) [2025-05-31 02:59:37.728658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=303][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.740562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.746808] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83076][T1_L0_G0][T1][YB42AC108714-000636671F1D63AF-0-0] [lt=49][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.746900] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83076][T1_L0_G0][T1][YB42AC108714-000636671F1D63AF-0-0] [lt=86][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.746921] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83076][T1_L0_G0][T1][YB42AC108714-000636671F1D63AF-0-0] [lt=14][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09ba80c820, local_datums:0x2b09ba80c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09ba80c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09ba80c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c4f3, hex: F48E88E969360600, int: 1748678377770740}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c4fb, hex: F48E88E969360600, int: 1748678377770740}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09ba80c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.747095] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83076][T1_L0_G0][T1][YB42AC108714-000636671F1D63AF-0-0] [lt=100][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.748341] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63AF-0-0] [lt=59] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:14124}, to_seq_no={branch:0, seq:8581}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382280} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.751175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.761656] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83097][T1_L0_G0][T1][YB42AC108714-000636671F1D63B0-0-0] [lt=51] get number(ret=0, service_type_=0, range=1, base_id=1748678377761429000, start_id=1748678377761429000, end_id=1748678377761429001) [2025-05-31 02:59:37.761832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.772651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.776289] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B1-0-0] [lt=48][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.776337] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B1-0-0] [lt=43][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.776357] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B1-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f3, hex: 1D1D89E969360600, int: 1748678377807133}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4fb, hex: 1D1D89E969360600, int: 1748678377807133}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c503, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c50b, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.776546] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B1-0-0] [lt=63][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.777793] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63B1-0-0] [lt=104] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7995}, to_seq_no={branch:0, seq:6435}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382281} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.781983] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:37.782022] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] temporary file module use no blocks [2025-05-31 02:59:37.782033] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:37.782048] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:37.782062] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525425963370, flush_io_finished_ret_:0, flush_io_finished_round_:19945, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:37.783143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.793691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.795507] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678377581947001, v:0}, new_max_version={val:1748678377581947001, v:0}, last_min_version={val:1748678377180728001, v:0}, last_max_version={val:1748678377180728001, v:0}, total_time_us=9404, delta_us=213556) [2025-05-31 02:59:37.795564] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678377581947001, v:0}, min_version={val:1748678377581947001, v:0}, max_version={val:1748678377581947001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 03:59:37.796879] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [36075][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:37.797371] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=96] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 03:59:37.808697] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [36075][T1_L0_G1][T1][YB42AC1087A7-00063665517F571C-0-0] [lt=58] init point thread id with(&point=0x55c0908fae80, idx_=3307, point=[thread id=36075, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=36075) [2025-05-31 02:59:37.811953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.813769] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=41][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af47f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382283} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3647} write_epoch=1, prev=this=0x2b0988ae68e0 trans_version={val:1748678317189600000, v:0} scn={val:1748678317189600000, v:0} tx_id={txid:11382204} prev=0x2b0988ae6728 next=(nil) modify_count=115 acc_checksum=191157785 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:15073} write_epoch=3, snapshot_version={val:1748678377812033000, v:0}, *this={this=0x2b097cccada0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae68e0 latest_compact_node=0x2b0980346330 max_trans_version={val:1748678317189600000, v:0} max_trans_id=11382204 max_elr_trans_version={val:1748678317189600000, v:0} max_elr_trans_id=11382204 latest_compact_ts=1748678256414153 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678317189600000, v:0} min_modify_scn={val:1748674855283998000, v:0}}) [2025-05-31 02:59:37.813975] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=202][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af47f0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382283} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3647} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407711343, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407711343, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678377812033000, v:0}, tx_id:{txid:11382283}, scn:{branch:0, seq:3644}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382283}, tx_desc:{this:0x2b094e970d20, tx_id:{txid:11382283}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678377812033, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678377846421, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:3647}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:37.814077] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=101][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382283}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3647}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:37.814096] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382283} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:37.814128] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=29][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.814143] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: 96B689E969360600, int: 1748678377846422}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: 96B689E969360600, int: 1748678377846422}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: A63A790D00000000, int: 226048678}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: A63A790D00000000, int: 226048678}]}) [2025-05-31 02:59:37.814253] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=55][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.815374] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63B5-0-0] [lt=107] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4965}, to_seq_no={branch:0, seq:3645}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382283} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.822218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.832460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.841941] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=11][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af4c48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382285} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:6633} write_epoch=1, prev=this=0x2b0988ae6d38 trans_version={val:1748678317209919000, v:0} scn={val:1748678317209919000, v:0} tx_id={txid:11382205} prev=0x2b0988ae6b80 next=(nil) modify_count=115 acc_checksum=3807362560 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:12244} write_epoch=3, snapshot_version={val:1748678377838603000, v:0}, *this={this=0x2b09802c7738 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae6d38 latest_compact_node=0x2b0980346400 max_trans_version={val:1748678317209919000, v:0} max_trans_id=11382205 max_elr_trans_version={val:1748678317209919000, v:0} max_elr_trans_id=11382205 latest_compact_ts=1748678256481655 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678317209919000, v:0} min_modify_scn={val:1748674855335943001, v:0}}) [2025-05-31 02:59:37.842059] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=114][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af4c48 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382285} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:6633} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407737061, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407737061, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678377838603000, v:0}, tx_id:{txid:11382285}, scn:{branch:0, seq:6415}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382285}, tx_desc:{this:0x2b094e9d3f80, tx_id:{txid:11382285}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678377840935, commit_ts:-1, finish_ts:-1, timeout_us:29999991, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678377871308, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:6633}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:37.842160] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=100][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382285}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:6633}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:37.842179] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382285} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:37.842210] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.842226] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: CD178AE969360600, int: 1748678377871309}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: CD178AE969360600, int: 1748678377871309}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: B196BB0900000000, int: 163288753}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: B196BB0900000000, int: 163288753}]}) [2025-05-31 02:59:37.842334] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=57][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.845989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.846233] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83089][T1_L0_G0][T1][YB42AC108714-000636671F1D63B7-0-0] [lt=45] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8753}, to_seq_no={branch:0, seq:6416}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382285} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.856251] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.867086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.872665] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:37.874735] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63B9-0-0] [lt=43][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.874758] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63B9-0-0] [lt=20][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.874774] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63B9-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b095c40c820, local_datums:0x2b095c40c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b095c40c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b095c40c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4f2, hex: 99948AE969360600, int: 1748678377903257}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c4fa, hex: 99948AE969360600, int: 1748678377903257}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c502, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b095c40c50a, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.875001] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83092][T1_L0_G0][T1][YB42AC108714-000636671F1D63B9-0-0] [lt=63][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.877725] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83059][T1_L0_G0][T1][YB42AC108714-000636671F1D63B9-0-0] [lt=37] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10674}, to_seq_no={branch:0, seq:5883}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382290} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.880727] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.894761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.897272] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BB-0-0] [lt=11][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.897306] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BB-0-0] [lt=30][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.897325] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BB-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: DBF48AE969360600, int: 1748678377927899}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: DBF48AE969360600, int: 1748678377927899}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:37.897494] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BB-0-0] [lt=69][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.900792] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC108714-000636671F1D63BB-0-0] [lt=27] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8944}, to_seq_no={branch:0, seq:4895}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382291} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.914937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.916505] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:139) [81658][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-05-31 02:59:37.920523] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=75] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:37.927021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.932607] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BD-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:37.932657] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BD-0-0] [lt=46][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.932678] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BD-0-0] [lt=12][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f2, hex: B7768BE969360600, int: 1748678377961143}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4fa, hex: B7768BE969360600, int: 1748678377961143}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c502, hex: 00606F0A00000000, int: 175071232}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c50a, hex: 00606F0A00000000, int: 175071232}]}) [2025-05-31 02:59:37.932813] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63BD-0-0] [lt=65][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.935763] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63BD-0-0] [lt=39] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:10797}, to_seq_no={branch:0, seq:4746}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382292} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.938648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=692][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.946154] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83074][T1_L0_G0][T1][YB42AC108714-000636671F1D63BE-0-0] [lt=12] get number(ret=0, service_type_=0, range=1, base_id=1748678377945374000, start_id=1748678377945374000, end_id=1748678377945374001) [2025-05-31 02:59:37.948780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 03:59:37.950438] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [36117][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:37.950941] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89] worker thread created(tenant_->id()=1, group_id_=1, token=8) [2025-05-31 02:59:37.957833] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=31][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af5598 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382293} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2606} write_epoch=1, prev=this=0x2b0980339900 trans_version={val:1748677892106370000, v:0} scn={val:1748677892106370000, v:0} tx_id={txid:11381626} prev=0x2b0988a88450 next=(nil) modify_count=11 acc_checksum=2662266613 version=1748674823531525 type=1 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:25566} write_epoch=0, snapshot_version={val:1748678377953865000, v:0}, *this={this=0x2b09b31623c0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=0 list_head=0x2b0980339900 latest_compact_node=0x2b0980339900 max_trans_version={val:1748678317314272000, v:0} max_trans_id=11382209 max_elr_trans_version={val:1748678317314272000, v:0} max_elr_trans_id=11382209 latest_compact_ts=1748677892115980 last_compact_cnt=11 total_trans_node_cnt=12 max_modify_scn={val:1748677892106370000, v:0} min_modify_scn={val:1748675827281255000, v:0}}) [2025-05-31 02:59:37.957982] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=145][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af5598 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382293} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:2606} write_epoch=1, ctx={this:0x2b091c3cf1a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407855347, tablet_id:{id:309}, table_iter:0x2b091c3cd6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407855347, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678377953865000, v:0}, tx_id:{txid:11382293}, scn:{branch:0, seq:2603}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382293}, tx_desc:{this:0x2b099129a560, tx_id:{txid:11382293}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678377957494, commit_ts:-1, finish_ts:-1, timeout_us:29999992, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678377990690, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904ce2b50, mem_ctx:0x2b0904ce40d0, tx_scn:{branch:0, seq:2606}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:37.958078] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=94][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382293}, data:0x2b091c3cc0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:2606}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:37.958096] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant local data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382293} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:37.958127] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:37.958142] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09a0e0c820, local_datums:0x2b09a0e0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09a0e0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 17, flag: 0, null: 0, ptr: 0x2b09a0e0c4e1, hex: 74656E616E74206C6F63616C2064617461, cstr: tenant local data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c4f2, hex: 23EA8BE969360600, int: 1748678377990691}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c4fa, hex: 23EA8BE969360600, int: 1748678377990691}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c502, hex: 00203F0100000000, int: 20914176}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09a0e0c50a, hex: 00203F0100000000, int: 20914176}]}) [2025-05-31 02:59:37.958263] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83077][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:37.962117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.962497] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63BF-0-0] [lt=45] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5414}, to_seq_no={branch:0, seq:2604}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382293} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:37.967352] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44AA-0-0] [lt=19][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:37.972438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.980056] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81060][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=38] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:37.983967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:37.993070] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:37.993159] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66] get_max_applied_scn(scn={val:1748678377859162000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:11, palf_committed_end_lsn_:{lsn:2137482647}, palf_committed_end_scn_:{val:1748678377986462000, v:0}, last_check_scn_:{val:1748678377986462001, v:0}, max_applied_cb_scn_:{val:1748678377859162000, v:0}}) [2025-05-31 02:59:37.993193] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:1979154949}, scn={val:1748659848819527001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1748659848671767, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:70440975, ret_code_:0}}, next_to_submit_lsn_:{lsn:1979154949}, next_to_submit_scn_:{val:1748659848819527001, v:0}, base_lsn_:{lsn:1543409664}, base_scn_:{val:1748584937774737007, v:0}, iterator_:{iterator_impl:{buf_:0x2b094e405000, next_round_pread_size:3678208, curr_read_pos:2154, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2154, log_storage_:{start_lsn:{lsn:1979152795}, end_lsn:{lsn:1979154949}, read_buf:{buf_len_:3747840, buf_:0x2b094e405000}, block_size:67104768, log_storage_:0x2b09131ebbf0, log_storage_:{log_tail:{lsn:2137482647}, readable_log_tail:{lsn:2137482647}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2080247808}, min_scn:{val:1748674078499979001, v:0}, curr_block_id:31, palf_id:1}, block_mgr:{log_dir:"/data/ob/data/clog/tenant_1/1/log", dir_fd:124, min_block_id:8, max_block_id:32, curr_writable_block_id:31}, logical_block_size_:67104768, curr_block_writable_size_:9869929, block_header_serialize_buf_:0x2b09131ec8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1748659848819527000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:984, scn_:{val:1748659848819527000, v:0}, data_checksum:2144344518, flag:43749}}, init_mode_version:0, accumulate_checksum:2148015145, curr_entry_is_padding:0, padding_entry_size:1016, padding_entry_scn:{val:1748659848819527000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:14452, miss_cnt:121, cache_read_size:7752276, hit_ratio:"9.916969738557606107e-01"}, cold_cache_stat:{hit_cnt:0, miss_cnt:121, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:221, read_io_size:431796224, read_disk_cost_ts:8854310, start_lsn:{lsn:1543409664}}}}}}) [2025-05-31 02:59:37.994362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.001995] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63C3-0-0] [lt=13][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:38.002033] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63C3-0-0] [lt=34][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.002051] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63C3-0-0] [lt=10][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: 63978CE969360600, int: 1748678378035043}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: 63978CE969360600, int: 1748678378035043}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:38.002353] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63C3-0-0] [lt=138][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.003458] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C3-0-0] [lt=44] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:4494}, to_seq_no={branch:0, seq:2148}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382295} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.004743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.011097] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83089][T1_L0_G0][T1][YB42AC108714-000636671F1D63C4-0-0] [lt=71] get number(ret=0, service_type_=0, range=1, base_id=1748678378010334000, start_id=1748678378010334000, end_id=1748678378010334001) [2025-05-31 02:59:38.015113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.024441] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=47][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af5e30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382296} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4165} write_epoch=1, prev=this=0x2b09802c7c30 trans_version={val:1748674855578654000, v:0} scn={val:1748674855578654000, v:0} tx_id={txid:11377462} prev=0x2b09b3162c20 next=(nil) modify_count=1 acc_checksum=1506457104 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:25777} write_epoch=3, snapshot_version={val:1748678378021099000, v:0}, *this={this=0x2b09b3162ac0 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b09802c7c30 latest_compact_node=(nil) max_trans_version={val:1748678317375328000, v:0} max_trans_id=11382212 max_elr_trans_version={val:1748678317375328000, v:0} max_elr_trans_id=11382212 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1748674855578654000, v:0} min_modify_scn={val:1748674855578654000, v:0}}) [2025-05-31 02:59:38.024513] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=70][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af5e30 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382296} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:4165} write_epoch=1, ctx={this:0x2b091c8d31a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407921691, tablet_id:{id:309}, table_iter:0x2b091c8d16d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407921691, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678378021099000, v:0}, tx_id:{txid:11382296}, scn:{branch:0, seq:4162}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382296}, tx_desc:{this:0x2b098d94a600, tx_id:{txid:11382296}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678378024000, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678378055975, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c99ed0, mem_ctx:0x2b0904c9b450, tx_scn:{branch:0, seq:4165}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:38.024676] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=161][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382296}, data:0x2b091c8d00c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:4165}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:38.024697] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=19][errcode=0] primary key duplication conflict(key={"BIGINT":1003},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382296} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:38.024773] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=72][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.024790] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=9][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c120c820, local_datums:0x2b09c120c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c120c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c120c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c4f1, hex: 28E98CE969360600, int: 1748678378055976}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c4f9, hex: 28E98CE969360600, int: 1748678378055976}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c501, hex: 0060090000000000, int: 614400}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c120c509, hex: 0060090000000000, int: 614400}]}) [2025-05-31 02:59:38.024985] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=133][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.025469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=144][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.027721] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83061][T1_L0_G0][T1][YB42AC108714-000636671F1D63C5-0-0] [lt=105] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8379}, to_seq_no={branch:0, seq:4163}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382296} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.036633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.048073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.051146] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137483793}], scn_array=[cnt:1, {val:1748678378033849000, v:0}], curr_size=561, accum_size=13684, time_cost=13998) [2025-05-31 02:59:38.058723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.064409] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=9][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b097cd46160 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382297} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3104} write_epoch=1, prev=this=0x2b0988ae8030 trans_version={val:1748678317397311000, v:0} scn={val:1748678317397311000, v:0} tx_id={txid:11382213} prev=0x2b0988ae7e78 next=(nil) modify_count=115 acc_checksum=2527985586 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:17543} write_epoch=3, snapshot_version={val:1748678378058809000, v:0}, *this={this=0x2b09b3162d68 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2b0988ae8030 latest_compact_node=0x2b0980346598 max_trans_version={val:1748678317397311000, v:0} max_trans_id=11382213 max_elr_trans_version={val:1748678317397311000, v:0} max_elr_trans_id=11382213 latest_compact_ts=1748678256725728 last_compact_cnt=113 total_trans_node_cnt=116 max_modify_scn={val:1748678317397311000, v:0} min_modify_scn={val:1748674855636178000, v:0}}) [2025-05-31 02:59:38.064467] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=57][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b097cd46160 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382297} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:3104} write_epoch=1, ctx={this:0x2b091c4551a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678407961007, tablet_id:{id:309}, table_iter:0x2b091c4536d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678407961007, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678378058809000, v:0}, tx_id:{txid:11382297}, scn:{branch:0, seq:3101}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382297}, tx_desc:{this:0x2b098b352780, tx_id:{txid:11382297}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678378060119, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678378094407, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904cf81d0, mem_ctx:0x2b0904cf9750, tx_scn:{branch:0, seq:3104}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:38.064581] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=110][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382297}, data:0x2b091c4520c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:3104}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:38.064599] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=17][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant slog data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382297} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:38.064629] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.064644] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b0983a0c820, local_datums:0x2b0983a0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b0983a0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b0983a0c4e1, hex: 74656E616E7420736C6F672064617461, cstr: tenant slog data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c4f1, hex: 487F8DE969360600, int: 1748678378094408}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c4f9, hex: 487F8DE969360600, int: 1748678378094408}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c501, hex: 00B0380000000000, int: 3715072}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b0983a0c509, hex: 00B0380000000000, int: 3715072}]}) [2025-05-31 02:59:38.064837] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83078][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=139][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.066493] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63C7-0-0] [lt=55] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:8802}, to_seq_no={branch:0, seq:3102}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382297} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.070284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.072938] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:38.080701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.091093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.097676] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=48] current monitor number(seq_=-1) [2025-05-31 02:59:38.097773] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=22] gc stale ls task succ [2025-05-31 02:59:38.100305] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83083][T1_L0_G0][T1][YB42AC108714-000636671F1D63CA-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1748678378099107000, start_id=1748678378099107000, end_id=1748678378099107001) [2025-05-31 02:59:38.101674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.101976] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=25] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:128K/27:501931 disk:128K/27:501931 t1m2:128K/27:501931 t1m2g0:128K/27:501931 |q0: 0 |q1: 0 [2025-05-31 02:59:38.111499] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83083][T1_L0_G0][T1][YB42AC108714-000636671F1D63CB-0-0] [lt=32][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:38.111537] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83083][T1_L0_G0][T1][YB42AC108714-000636671F1D63CB-0-0] [lt=34][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.111555] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83083][T1_L0_G0][T1][YB42AC108714-000636671F1D63CB-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09c8a0c820, local_datums:0x2b09c8a0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c4bc, hex: EB03000000000000, int: 1003}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09c8a0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09c8a0c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c4f1, hex: 913C8EE969360600, int: 1748678378142865}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c4f9, hex: 913C8EE969360600, int: 1748678378142865}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c501, hex: 00C0BC0000000000, int: 12369920}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09c8a0c509, hex: 00C0BC0000000000, int: 12369920}]}) [2025-05-31 02:59:38.112095] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83083][T1_L0_G0][T1][YB42AC108714-000636671F1D63CB-0-0] [lt=133][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.112641] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.116192] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83107][T1_L0_G0][T1][YB42AC108714-000636671F1D63CB-0-0] [lt=128] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:7670}, to_seq_no={branch:0, seq:4032}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382299} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.122632] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:38.123183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.126061] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:38.126325] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:38.126390] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:38.133213] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=31] [APPLY STAT CB APPEND COST TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=163, this=0x2b0913fa6280, extra_info="") [2025-05-31 02:59:38.133254] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=40] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=6015, this=0x2b0913fa64b0, extra_info="") [2025-05-31 02:59:38.133266] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=0, this=0x2b0913fa66e0, extra_info="") [2025-05-31 02:59:38.133273] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=-898, this=0x2b0913fa6910, extra_info="") [2025-05-31 02:59:38.133280] INFO [LIB] stat (utility.h:1211) [81932][T1_ApplySrv0][T1][YB42AC1087A6-000636654C3A38F9-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=67, stat_interval=5000000, avg (count/cost)=5280, this=0x2b0913fa6b40, extra_info="") [2025-05-31 02:59:38.133976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.143052] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:38.145123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.148561] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:937) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=16][errcode=-5024] find double insert node(ret=-5024, node=this=0x2b0988af60b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382300} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:9057} write_epoch=1, prev=this=0x2b0988a89138 trans_version={val:1748677892366043000, v:0} scn={val:1748677892366043000, v:0} tx_id={txid:11381633} prev=0x2b0988a88f80 next=(nil) modify_count=9 acc_checksum=2035094073 version=1748674823531525 type=0 flag=4 snapshot_barrier=1748678377378499000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:43911} write_epoch=3, snapshot_version={val:1748678378140456000, v:0}, *this={this=0x2b09802c7d90 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x2b0988a89138 latest_compact_node=0x2b0980319770 max_trans_version={val:1748678317458790000, v:0} max_trans_id=11382216 max_elr_trans_version={val:1748678317458790000, v:0} max_elr_trans_id=11382216 latest_compact_ts=1748677042082843 last_compact_cnt=5 total_trans_node_cnt=10 max_modify_scn={val:1748677892366043000, v:0} min_modify_scn={val:1748675827437572000, v:0}}) [2025-05-31 02:59:38.148637] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:999) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=73][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2b0988af60b0 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:11382300} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1748674823531525 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:91} seq_no={branch:0, seq:9057} write_epoch=1, ctx={this:0x2b091bf511a0, ls_id:{id:1}, ls:0x2b0913f54150, branch:0, timeout:1748678408037274, tablet_id:{id:309}, table_iter:0x2b091bf4f6d8, table_version:1748168832130112, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1748678408037274, tx_lock_timeout_us:-1, snapshot:{version:{val:1748678378140456000, v:0}, tx_id:{txid:11382300}, scn:{branch:0, seq:9054}}, tx_table_guards:{tx_table_guard:{tx_table:0x2b0913f5fb10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:11382300}, tx_desc:{this:0x2b094e9f8640, tx_id:{txid:11382300}, state:1, addr:"172.16.135.20:2882", tenant_id:1, session_id:1, assoc_session_id:0, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:-1, active_ts:1748678378148106, commit_ts:-1, finish_ts:-1, timeout_us:29999988, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1748168748, cluster_version:17180067073, seq_base:1748678378171415, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:0, v:0}, ref:1}, tx_ctx:0x2b0904c803d0, mem_ctx:0x2b0904c81950, tx_scn:{branch:0, seq:9057}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, is_standby_read:false, lock_wait_start_ts:0, is_inited:true}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-05-31 02:59:38.148735] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:333) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=97][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:11382300}, data:0x2b091bf4e0c0, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1748674823531525, seq_no:{branch:0, seq:9057}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-05-31 02:59:38.148753] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:314) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1004},{"VARCHAR":"172.16.135.20", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":11},{"VARCHAR":"tenant meta data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:11382300} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-05-31 02:59:38.148784] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.148991] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=198][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: EC03000000000000, int: 1004}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E74206D6574612064617461, cstr: tenant meta data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f1, hex: 18AC8EE969360600, int: 1748678378171416}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f9, hex: 18AC8EE969360600, int: 1748678378171416}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c501, hex: 00908A0000000000, int: 9080832}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c509, hex: 00908A0000000000, int: 9080832}]}) [2025-05-31 02:59:38.149332] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=243][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.156285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.164218] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83088][T1_L0_G0][T1][YB42AC108714-000636671F1D63CD-0-0] [lt=67] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:23165}, to_seq_no={branch:0, seq:9055}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382300} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.167579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.176799] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO [ratelimit] time: 1748678378176796, bytes: 7141552498, bw: 0.775810 MB/s, add_ts: 1001073, add_bytes: 814369 [2025-05-31 02:59:38.178841] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.185137] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [81929][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=44] LogLoopThread round_cost_time(us)(round_cost_time=67) [2025-05-31 02:59:38.189109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.200677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.211454] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=104][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137487469}], scn_array=[cnt:1, {val:1748678378188997000, v:0}], curr_size=589, accum_size=17388, time_cost=16431) [2025-05-31 02:59:38.211786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.216948] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1312) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CF-0-0] [lt=40][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-05-31 02:59:38.216980] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4643) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CF-0-0] [lt=28][errcode=-5024] fail to set memtable(ret=-5024) [2025-05-31 02:59:38.216999] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5667) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CF-0-0] [lt=11][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2b09adc0c820, local_datums:0x2b09adc0c820}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4bc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 13, flag: 0, null: 0, ptr: 0x2b09adc0c4c4, hex: 3137322E31362E3133352E3230, cstr: 172.16.135.20}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d1, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4d9, hex: 0B00000000000000, int: 11}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2b09adc0c4e1, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f0, hex: D27C8FE969360600, int: 1748678378224850}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c4f8, hex: D27C8FE969360600, int: 1748678378224850}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c500, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2b09adc0c508, hex: 0000000000000000, int: 0}]}) [2025-05-31 02:59:38.217219] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3241) [83067][T1_L5_G0][T1][YB42AC108714-000636671F1D63CF-0-0] [lt=68][errcode=-5024] failed to write row(ret=-5024) [2025-05-31 02:59:38.219451] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:851) [83085][T1_L0_G0][T1][YB42AC108714-000636671F1D63CF-0-0] [lt=98] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:30633}, to_seq_no={branch:0, seq:24174}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1748168832130112 max_table_version=1748168832130112 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:11382301} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-05-31 02:59:38.224017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.230187] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1748678378230183, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000924, add_bytes: 0 [2025-05-31 02:59:38.235833] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.246238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.259064] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.269526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.273212] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:38.281859] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.292200] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=134][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.295989] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81065][TimerWK3_DumpLSLoc][T0][YB42AC1087A6-0006366545EA56F8-0-0] [lt=24] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678373709029, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.167:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:38.296055] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81065][TimerWK3_DumpLSLoc][T0][YB42AC1087A6-0006366545EA56F8-0-0] [lt=63] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678373711045, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:38.296082] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1221) [81065][TimerWK3_DumpLSLoc][T0][YB42AC1087A6-0006366545EA56F8-0-0] [lt=21] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1748168748}, renew_time:1748674295940278, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:13}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1006}, cluster_id:1748168748}, renew_time:1748674225831439, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1748168748}, renew_time:1748674215812166, replica_locations:[{server:"172.16.135.166:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:15}, {server:"172.16.135.30:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1748168748}, renew_time:1748678373716002, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:12}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1004}, cluster_id:1748168748}, renew_time:1748678373716008, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:4}, {server:"172.16.135.168:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1005}, cluster_id:1748168748}, renew_time:1748674285931073, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:3}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1748168748}, renew_time:1748674571316283, replica_locations:[{server:"172.16.135.10:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:11}, {server:"172.16.135.20:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-05-31 02:59:38.302419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.313036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.323542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.324171] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:38.336444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.347620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.358285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.359525] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.359586] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.359622] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.359912] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.361185] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207E-0-0] [lt=33] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:38.361213] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207E-0-0] [lt=19] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=30) [2025-05-31 02:59:38.368720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.379148] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.381766] INFO pn_ratelimit (group.c:653) [81063][TimerWK1_IngressService][T0][Y0-0000000000000000-0-0] [lt=35] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-31 02:59:38.391116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.398032] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=50] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:38.398058] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81060][TimerWK0_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:38.398484] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F54-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:38.398817] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F54-0-0] [lt=327] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:38.402279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.415634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.425902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.436267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.446653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.458595] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=70][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.469122] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.478043] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=73] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=581936, replace_num=15728) [2025-05-31 02:59:38.478081] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:38.479510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.490324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.500633] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.511393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.520347] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=82, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:38.520403] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=31, avg_log_batch_cnt=1, total_group_log_size=18656, avg_group_log_size=601) [2025-05-31 02:59:38.520475] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=66] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=19724) [2025-05-31 02:59:38.520517] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=31, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:38.522101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.522119] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:38.522242] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4642, iops:30, norm_iops:8, bw:137632, [delay/us]:prepare:19, schedule:580, submit:292, rt:3254, total:4146) [2025-05-31 02:59:38.522280] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=8, iops=8, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:38.522732] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=432] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4642, avg_iops:30, avg_bw:137598, [delay/us]: prepare:19, schedule:580, submit:292, device:3254, total:4146) [2025-05-31 02:59:38.522773] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:38.522804] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4046, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:308, total:322) [2025-05-31 02:59:38.522831] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:38.522976] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=138] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4046, [delay/us]: prepare:0, schedule:0, submit:600000000, device:308, total:322) [2025-05-31 02:59:38.523024] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:38.523079] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:38.523127] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:38.523138] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:38.523157] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:38.523170] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SERVER] net_in=831kB/s, net_out=378kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:38.523203] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] successfully alloc block(macro_id={[ver=1,mode=0,seq=399473][2nd=2517]}) [2025-05-31 02:59:38.526390] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=18656) [2025-05-31 02:59:38.526904] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:38.527127] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=31, avg_log_gen_to_freeze_time=429, avg_log_gen_to_submit_time=530, avg_log_submit_to_flush_time=5192, avg_log_submit_to_slide_time=5743) [2025-05-31 02:59:38.527234] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=18656, hit_cnt=31, read_cnt=31, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:38.532473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.543025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.553464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.564444] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.574873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.580300] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1748678378580262], gts:1748678378580011000, latest_srr:[mts=1748678378580262]}) [2025-05-31 02:59:38.580346] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=43] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:38.585084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=81][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.589494] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA98-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:38.590941] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592069] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592162] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592171] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592199] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592211] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.592235] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:38.596194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.606722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.617133] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.626621] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:38.628487] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.639428] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.642882] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81743][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:38.643198] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81740][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=69] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:38.652479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.663374] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=107][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.673583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.682742] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3904-0-0] [lt=48][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525427864911, cache_obj->added_lc()=false, cache_obj->get_object_id()=53740, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:38.682990] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-05-31 02:59:38.683903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.685720] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3906-0-0] [lt=36][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:38.686829] WDIAG get_refactored (ob_hashtable.h:1120) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3908-0-0] [lt=26][errcode=-4201] hashtable not init [2025-05-31 02:59:38.687191] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3909-0-0] [lt=110] get number(ret=0, service_type_=0, range=1, base_id=1748678378685935000, start_id=1748678378685935000, end_id=1748678378685935001) [2025-05-31 02:59:38.690714] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=47] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-31 02:59:38.690753] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [81743][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=24] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=989) [2025-05-31 02:59:38.693194] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=16426) [2025-05-31 02:59:38.697316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.698041] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38166][T1_L0_G5][T1][YB42AC10870A-0006366715B99E47-0-0] [lt=108] get number(ret=0, service_type_=0, range=1, base_id=1748678378697082000, start_id=1748678378697082000, end_id=1748678378697082001) [2025-05-31 02:59:38.703500] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [81740][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=25] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:969, free_obj_hold_cnt:161, allocator used:4339200, allocator total:4634880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:153, free_obj_hold_cnt:247, allocator used:26236400, allocator total:26236400}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=30871280, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:48, free_obj_hold_cnt:14, allocator used:126976, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=1122) [2025-05-31 02:59:38.707598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.718147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.727701] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A390A-0-0] [lt=28][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525427909870, cache_obj->added_lc()=false, cache_obj->get_object_id()=53741, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:38.727903] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-46510080, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:38.729730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.739939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.750085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.760238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.775569] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20E3-0-0] [lt=37] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:38.778067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.790552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.799120] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83001][T1_TenantWeakRe][T1][YB42AC1087A6-00063665494DFC32-0-0] [lt=81] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.726568365097045898e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=286, Throttle Trigger(MB)=171, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.586889580866345113e+03) [2025-05-31 02:59:38.801077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.803703] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT WRITE LOG COST TIME](cur_stat_count=27, stat_interval=1000000, avg (count/cost)=4566, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:38.803753] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=27, stat_interval=1000000, avg (count/cost)=546, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:38.803972] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=27, stat_interval=1000000, avg (count/cost)=27, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:38.803997] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=27, stat_interval=1000000, avg (count/cost)=6304, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:38.804032] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=27, stat_interval=1000000, avg (count/cost)=68, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:38.804288] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=119] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678378595095000, v:0}, new_max_version={val:1748678378595095000, v:0}, last_min_version={val:1748678378396686000, v:0}, last_max_version={val:1748678378396686000, v:0}, total_time_us=7196, delta_us=209190) [2025-05-31 02:59:38.804341] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678378595095000, v:0}, min_version={val:1748678378595095000, v:0}, max_version={val:1748678378595095000, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:38.811538] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.821780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.832042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=76][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.844794] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83061][T1_L0_G0][T1][YB42AC10871E-000636654E089A54-0-0] [lt=43][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525428026961, cache_obj->added_lc()=false, cache_obj->get_object_id()=53742, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:38.845555] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.856067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.867026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.877445] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.883177] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81063][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-05-31 02:59:38.883302] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:38.883483] WDIAG read_multi_row (ob_table_access_helper.h:238) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-05-31 02:59:38.885738] INFO delete_row (ob_table_access_helper.h:339) [82983][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=45] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1748678378883530) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1748678378883530) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-05-31 02:59:38.889660] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-05-31 02:59:38.889696] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=27] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:59:38.889820] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=112] [TenantFreezer] oldest frozen memtable(list="") [2025-05-31 02:59:38.890731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=98][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.901420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.911614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.922020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.922700] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81065][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=94] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:38.931717] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81060][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-42315776, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:38.934054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=108][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.935387] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [81740][T1_TimerWK0_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=197] clear dtl interm result cost(us)(clear_cost=3477, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-31 02:59:38.936310] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137491069}], scn_array=[cnt:1, {val:1748678378796912001, v:0}], curr_size=122, accum_size=1865, time_cost=10815) [2025-05-31 02:59:38.944837] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.957242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=129][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.957294] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-31 02:59:38.957322] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:38.957391] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [32846][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=44] worker thread exit(tenant_->id()=1, workers_.get_size()=8) [2025-05-31 02:59:38.969178] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.970554] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44AB-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:38.980391] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:38.980784] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81063][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=28] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:38.991112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.001679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=53][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.002817] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [81490][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=40] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=1295) [2025-05-31 02:59:39.009807] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:39.012983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.023539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=101][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.033797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.046495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.056771] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.067409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=154][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.068364] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] ====== check clog disk timer task ====== [2025-05-31 02:59:39.068409] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [81744][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1590) [2025-05-31 02:59:39.077730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.083397] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-05-31 02:59:39.087954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.099821] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=45] current monitor number(seq_=-1) [2025-05-31 02:59:39.099911] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=20] gc stale ls task succ [2025-05-31 02:59:39.100017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.112544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.122831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.126349] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=124] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:39.127047] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:39.127181] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:39.132522] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-50704384, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:39.137304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.145299] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=47] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:39.148078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.159012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.169264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.179617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.182023] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=93] PNIO [ratelimit] time: 1748678379182019, bytes: 7141804767, bw: 0.239332 MB/s, add_ts: 1005223, add_bytes: 252269 [2025-05-31 02:59:39.183453] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1575) [26524][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA8-0-0] [lt=27] receive register_req from acceptor, ignore(self="172.16.135.166:2882", palf_id=1, child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678379218952, keepalive_ts:-1}) [2025-05-31 02:59:39.183482] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1594) [26524][T1_L0_G1][T1][YB42AC10871E-0006366546E85BA8-0-0] [lt=27] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.16.135.166:2882", child={server:"172.16.135.30:2882", region:"sys_region", register_time_us:1748678379218952, keepalive_ts:-1}, is_to_leader=true) [2025-05-31 02:59:39.190134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.201118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.204491] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=33] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:52K/12:550934 disk:52K/12:550934 t1m2:52K/12:550934 t1m2g0:52K/12:550934 |q0: 0 |q1: 0 [2025-05-31 02:59:39.212528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.222788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.233558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.235146] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1748678379235141, bytes: 0, bw: 0.000000 MB/s, add_ts: 1004958, add_bytes: 0 [2025-05-31 02:59:39.244468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=83][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.255242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.265738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.278085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.284046] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-05-31 02:59:39.288325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.300766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.311355] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.319228] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:171) [38166][T1_L0_G5][T1][YB42AC10871E-000636654DFA0E7E-0-0] [lt=31] handle gts request statistics(total_rt=1620, total_cnt=194, avg_rt=8.307692307692308376e+00, avg_cnt=1.939999999999999858e+01) [2025-05-31 02:59:39.324238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.333365] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=100] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:39.335004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.335222] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [82981][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=28] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-31 02:59:39.345361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.355613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=94][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.358604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.358952] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=91] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.359518] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.359558] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.365944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.376236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.386520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.396804] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.407086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.421364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.431988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.442625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.452985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.459970] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [81538][Blacklist][T0][Y0-0000000000000000-0-0] [lt=38] blacklist_loop exec finished(cost_time=28, is_enabled=true, send_cnt=0) [2025-05-31 02:59:39.464647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=71][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.475037] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.484739] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-05-31 02:59:39.485242] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.495511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.505948] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.516231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.523519] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:158) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=24] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-05-31 02:59:39.525028] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:418) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=49] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-05-31 02:59:39.525124] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:215) [81742][T1_TimerWK1_FreInfoReload][T1][Y0-0000000000000000-0-0] [lt=20] inner load succ(latest_freeze_info={frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}, freeze_info_={latest_snapshot_gc_scn:{val:1748678372256477000, v:0}, frozen_statuses_.count():6, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067073}, {frozen_version:{val:1748239200899758000, v:0}, schema_version:1748172424867656, data_version:17180067073}, {frozen_version:{val:1748325602484325000, v:0}, schema_version:1748314009335712, data_version:17180067073}, {frozen_version:{val:1748498404535672000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748584804533724000, v:0}, schema_version:1748483419327608, data_version:17180067073}, {frozen_version:{val:1748671203414810000, v:0}, schema_version:1748658462187336, data_version:17180067073}]}) [2025-05-31 02:59:39.526832] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.529100] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=83, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:39.529156] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=9, avg_log_batch_cnt=1, total_group_log_size=3876, avg_group_log_size=430) [2025-05-31 02:59:39.529179] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=3876) [2025-05-31 02:59:39.529194] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:39.534372] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=85] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:39.534429] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:39.534585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:1, hold_mem:8192 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, norm_iops:2, bw:40464, [delay/us]:prepare:12, schedule:1064, submit:123, rt:3798, total:4999) [2025-05-31 02:59:39.534607] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=4, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31433536, free_req_cnt=49999, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:39.534673] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40476, [delay/us]: prepare:12, schedule:1064, submit:123, device:3798, total:4999) [2025-05-31 02:59:39.534687] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:39.534714] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4047, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:425, total:449) [2025-05-31 02:59:39.534739] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:39.534768] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4048, [delay/us]: prepare:0, schedule:0, submit:600000000, device:425, total:449) [2025-05-31 02:59:39.534785] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:39.534798] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:39.534806] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:39.534814] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:39.534822] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:39.534830] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=231kB/s, net_out=142kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:39.534850] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=399474][2nd=2389]}) [2025-05-31 02:59:39.535953] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2025-05-31 02:59:39.536835] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [36117][T1_L0_G1][T1][YB42AC1087A7-00063665517F5738-0-0] [lt=4158] init point thread id with(&point=0x55c0908fb900, idx_=3349, point=[thread id=36117, timeout ts=19:00:00.0, last click point="(null):(null):0", last click ts=19:00:00.0], thread_id=36117) [2025-05-31 02:59:39.537341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=128][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.537819] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=3876) [2025-05-31 02:59:39.539081] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=14, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=6918, avg_log_submit_to_slide_time=7064) [2025-05-31 02:59:39.540702] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3876, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:39.547828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.558296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.569081] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.581099] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=115][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.585991] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [81545][TsMgr][T1][Y0-0000000000000000-0-0] [lt=32] ts response task statistics(alloc_count=487250, free_count=243624) [2025-05-31 02:59:39.586030] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:39.586083] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=4, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:39.586154] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=69] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1748678379083966], gts:1748678379121549000, latest_srr:[mts=1748678379586046]}) [2025-05-31 02:59:39.590197] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.591577] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.591677] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.591720] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.591746] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.591949] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.594492] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.594569] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:39.604109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.606763] INFO [CLOG] is_in_sync (ob_log_handler.cpp:692) [83067][T1_L5_G0][T1][YB42AC108714-000636671E2D3A7A-0-0] [lt=42] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-05-31 02:59:39.613754] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83089][T1_L0_G0][T1][YB42AC10870A-0006366724D8CA99-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:39.615966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.626156] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.630227] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:39.636991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.645558] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81743][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=32] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:39.647305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.659364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.672029] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.682269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.685714] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A390C-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525428867882, cache_obj->added_lc()=false, cache_obj->get_object_id()=53743, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:39.686784] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-05-31 02:59:39.692742] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.703109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.713443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=87][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.726112] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.733060] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [83067][T1_L5_G0][T1][YB42AC1087A8-000636672F8E3F60-0-0] [lt=32] get number(ret=0, service_type_=0, range=1, base_id=1748678379729093000, start_id=1748678379729093000, end_id=1748678379729093001) [2025-05-31 02:59:39.733819] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [35611][T1_L0_G1][T1][YB42AC1087A7-00063665517F573A-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=3310) [2025-05-31 02:59:39.736515] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83079][T1_L0_G0][T1][YB42AC1087A6-000636654C3A390E-0-0] [lt=46][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525428918622, cache_obj->added_lc()=false, cache_obj->get_object_id()=53744, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:39.737067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.737800] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:39.750264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.751220] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1421) [81664][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=41] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-05-31 02:59:39.758548] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1748678369751146, end_time=1748678379758544, dag_status={}) [2025-05-31 02:59:39.758589] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=37] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758606] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758670] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758679] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758686] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758693] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758700] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758707] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758714] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-31 02:59:39.758724] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-31 02:59:39.758733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [81723][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-31 02:59:39.760757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.772012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=280][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.782359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.784817] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:39.784930] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=103] temporary file module use no blocks [2025-05-31 02:59:39.784944] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:39.784960] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:39.784976] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525427964432, flush_io_finished_ret_:0, flush_io_finished_round_:19947, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:39.793340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=171][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.803701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.814110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=161][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.819125] INFO [LIB] stat (utility.h:1211) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=100] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5691, this=0x2b09131efdb0, extra_info="") [2025-05-31 02:59:39.819180] INFO [LIB] stat (utility.h:1228) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1334, this=0x2b09049fd870, extra_info="") [2025-05-31 02:59:39.819513] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x2b09131e56a0, extra_info="palf_id:1") [2025-05-31 02:59:39.819539] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8705, this=0x2b09131e58d0, extra_info="palf_id:1") [2025-05-31 02:59:39.819561] INFO [LIB] stat (utility.h:1211) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=71, this=0x2b09131effe0, extra_info="") [2025-05-31 02:59:39.820107] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:331) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=160] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1, new_min_version={val:1748678379406978001, v:0}, new_max_version={val:1748678379406978001, v:0}, last_min_version={val:1748678379406978000, v:0}, last_max_version={val:1748678379406978000, v:0}, total_time_us=8036, delta_us=413125) [2025-05-31 02:59:39.820147] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1748678379406978001, v:0}, min_version={val:1748678379406978001, v:0}, max_version={val:1748678379406978001, v:0}, in_service=true, leader_epoch=11) [2025-05-31 02:59:39.820208] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:118) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1748678379611983000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1748678379820162}, version_delta=208222) [2025-05-31 02:59:39.825101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=112][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.831009] INFO [SHARE] adaptive_update_limit (ob_share_throttle_define.cpp:110) [83067][T1_L5_G0][T1][YB42AC10871E-0006366542D8A753-0-0] [lt=48] adaptive update(Tenant ID=1, Config Specify Resource Limit(MB)=511, TxShare Current Memory Limit(MB)=283, Holding Memory(MB)=134, Tenant Remain Memory(MB)=249, Usable Remain Memory(MB)=149, Last Update Limit Timestamp=1748678379830945, Is Updated=true) [2025-05-31 02:59:39.831071] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [83067][T1_L5_G0][T1][YB42AC10871E-0006366542D8A753-0-0] [lt=58] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=5.678568410873413086e+01, this=0x2b0904d7f208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=283, Throttle Trigger(MB)=170, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.674694851235618898e+03) [2025-05-31 02:59:39.836331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.847226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.857002] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:384) [38169][T1_L0_G5][T1][YB42AC1087A8-000636672FDF1F6B-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1748678379854283000, start_id=1748678379854283000, end_id=1748678379854283001) [2025-05-31 02:59:39.857468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.870423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.881121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.886710] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [81114][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=145] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=19329347584, fast_delete_log_mode=false) [2025-05-31 02:59:39.889361] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81065][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-05-31 02:59:39.891426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.901823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.912138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=109][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.922300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.922502] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [82992][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=120] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b09049642b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-31 02:59:39.925443] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=116] BlockGCTimerTask success(ret=0, cost_time_us=19, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.16.135.166:2882", log_dir:"/data/ob/data/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-31 02:59:39.935660] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1748168819230394, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:75, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:184142722586, recv_hp_rpc_cnt:75008, recv_np_rpc_cnt:879635, recv_lp_rpc_cnt:0, recv_mysql_cnt:47200, recv_task_cnt:2477, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=395 cnt[3]=0 cnt[4]=0 cnt[5]=1235060 cnt[6]=173 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:9895, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748658438691612}{group_id:12, queue_size:0, recv_req_cnt:34, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748666728589137}{group_id:28, queue_size:0, recv_req_cnt:10, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=6 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1748678306164536}{group_id:15, queue_size:0, recv_req_cnt:18, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1748660325353412}{group_id:1, queue_size:0, recv_req_cnt:544876, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:7, nesting_worker_cnt:0, token_change:1748678378957317}{group_id:10, queue_size:0, recv_req_cnt:104151, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658403928408}{group_id:20, queue_size:0, recv_req_cnt:4, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1748674318847032}{group_id:19, queue_size:0, recv_req_cnt:18403, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1748678371636599}{group_id:2, queue_size:0, recv_req_cnt:151187, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389041651}{group_id:9, queue_size:0, recv_req_cnt:101304, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658389367565}{group_id:27, queue_size:0, recv_req_cnt:108131, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748658497659328}{group_id:5, queue_size:0, recv_req_cnt:273091, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1748659849301913}, rpc_stat_info: pcode=0x51c:cnt=684 pcode=0x717:cnt=484 pcode=0x1502:cnt=250 pcode=0x710:cnt=174 pcode=0x523:cnt=100, token_change_ts:1748658388755592, tenant_role:1}) [2025-05-31 02:59:39.936687] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1023] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384809298, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84331833, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384832737, tenant_role:0}) [2025-05-31 02:59:39.937428] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=738] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1748658384799779, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:84832498, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1748658384835448, tenant_role:0}) [2025-05-31 02:59:39.938489] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1029][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.940674] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81064][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:39.948808] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.959155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=151][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.959261] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:499) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-05-31 02:59:39.959614] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:511) [33405][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=31] worker thread exit(tenant_->id()=1, workers_.get_size()=7) [2025-05-31 02:59:39.969507] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.973651] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [81675][T1_COMMONLSSe][T1][YB42AC1087A6-00063665518A44AC-0-0] [lt=87][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:39.980062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:39.980980] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [81065][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=30] skip inspect bad block(last_check_time=1748658385819089, last_macro_idx=-1) [2025-05-31 02:59:39.990325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.000603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.013290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.023507] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.035551] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.046367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.058875] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.070466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.081072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.089128] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1748678379586446], gts:1748678379635540000, latest_srr:[mts=1748678380089085]}) [2025-05-31 02:59:40.089707] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-05-31 02:59:40.092930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.100051] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [85844][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=49] current monitor number(seq_=-1) [2025-05-31 02:59:40.100123] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [83003][T1_LSArchiveMgr][T1][YB42AC1087A6-000636654259FC95-0-0] [lt=86] gc stale ls task succ [2025-05-31 02:59:40.103172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=95][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.117062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.122770] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:165) [83004][T1_ArcSeq][T1][YB42AC1087A6-000636654269FC95-0-0] [lt=34] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2025-05-31 02:59:40.125117] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:746) [81743][T1_TimerWK2_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=30] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=22603, get_mem_high()=24159150, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-05-31 02:59:40.127303] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [83012][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=48] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-31 02:59:40.129352] INFO do_work (ob_rl_mgr.cpp:708) [81448][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-31 02:59:40.129574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.130986] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=58] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-05-31 02:59:40.130993] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B0E-0-0] [lt=19] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:40.135612] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [81652][T1_Occam][T1][YB42AC1087A6-00063665422A3B10-0-0] [lt=28] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-31 02:59:40.140793] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.142144] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:40.145807] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=92] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:40.151179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=105][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.161472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.171917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.182288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.187520] INFO eloop_run (eloop.c:145) [81426][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1748678380187514, bytes: 7142024988, bw: 0.208871 MB/s, add_ts: 1005495, add_bytes: 220221 [2025-05-31 02:59:40.192764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.203201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=224][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.216674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=86][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.219289] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1748678380013964000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=205227, in_cluster_service=true, cluster_version={val:1748678379813240000, v:0}, min_cluster_version={val:1748678379813240000, v:0}, max_cluster_version={val:1748678379813240000, v:0}, get_cluster_version_err=0, cluster_version_delta=405951, cluster_service_master="172.16.135.166:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=195640, succ_cluster_heartbeat_count=13585, cluster_heartbeat_interval=50000, local_cluster_version={val:1748678009054758001, v:0}, local_cluster_delta=371164433, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-31 02:59:40.228294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.234079] INFO try_report (ob_tc_stat.cpp:35) [81581][][T0][Y0-0000000000000000-0-0] [lt=90] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1003m0:3,x,0 t1003m0gd:1,x,0 t1003m1:3,x,0 t1003m1gd:1,x,0 t1003m2:3,x,x t1003m2gd:1,x,0 t1004m0:3,x,0 t1004m0gd:1,x,0 t1004m1:3,x,0 t1004m1gd:1,x,0 t1004m2:3,x,x t1004m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1004m0g0:100,x,0 t1004m1g0:100,x,0 t1004m2g0:100,x,x t1003m0g0:100,x,0 t1003m1g0:100,x,0 t1003m2g0:100,x,x|stat: root:48K/11:1113989 disk:48K/11:1113989 t1m2:48K/11:1113989 t1m2g0:48K/11:1113989 |q0: 0 |q1: 0 [2025-05-31 02:59:40.241314] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=80][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.242753] INFO eloop_run (eloop.c:145) [81436][pnio2][T0][Y0-0000000000000000-0-0] [lt=33] PNIO [ratelimit] time: 1748678380242749, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0 [2025-05-31 02:59:40.246072] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137496735}], scn_array=[cnt:1, {val:1748678380220510000, v:0}], curr_size=589, accum_size=4122, time_cost=14798) [2025-05-31 02:59:40.248275] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [83001][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=11, start_service_tstamp_=1748659849952024, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-31 02:59:40.251627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=89][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.264562] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.267933] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137497324}], scn_array=[cnt:1, {val:1748678380245176000, v:0}], curr_size=548, accum_size=4670, time_cost=12516) [2025-05-31 02:59:40.274867] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.285193] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=82][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.290388] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=90] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-05-31 02:59:40.297528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.307747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.318108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.328342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.339621] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.343085] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81063][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:40.350143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.359000] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81547][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.359097] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81548][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.359327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81549][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.359379] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81551][RpcIO][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.360521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.370730] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:269) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207F-0-0] [lt=82] [RS_MGR] force set rs list(master_rs="172.16.135.166:2882") [2025-05-31 02:59:40.370779] INFO [SERVER] handle_heartbeat (ob_service.cpp:3867) [86213][T1_L0_G25][T1][YB42AC1087A6-00063665516A207F-0-0] [lt=39] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"172.16.135.166:2882", server_id:1, rs_addr:"172.16.135.166:2882", rs_server_status:1, epoch_id:11}, hb_response={zone:"zone1", server:"172.16.135.166:2882", sql_port:2881, build_version:"4.3.5.1_101000042025031818-b6d5706eb3d2c5f501c7fa646ddbf32f3dc87069(Mar 18 2025 18:12:30)", start_service_time:1748658531155476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=52) [2025-05-31 02:59:40.372275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=237][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.382738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.394483] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.399572] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:446) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=36] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:40.399600] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:258) [81064][TimerWK2_LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1748674258961328, refresh_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_ret=0) [2025-05-31 02:59:40.400005] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:260) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F55-0-0] [lt=313] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:40.400080] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:285) [81450][SerScheQueue0][T0][YB42AC1087A6-00063665492A0F55-0-0] [lt=70] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}, schema_info={schema_version:1748675035279216, tenant_id:1003, sequence_id:12094627905642}) [2025-05-31 02:59:40.400328] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=1998, gc_interval=10000000) [2025-05-31 02:59:40.400382] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1584) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=26] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-05-31 02:59:40.400403] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] gc_check_member_list_ cost time(ret=0, time_us=48) [2025-05-31 02:59:40.400415] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] execute_gc cost time(ret=0, time_us=1) [2025-05-31 02:59:40.400427] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-31 02:59:40.400440] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-31 02:59:40.400451] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=12) [2025-05-31 02:59:40.400459] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-05-31 02:59:40.400466] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-31 02:59:40.400486] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [81945][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-31 02:59:40.404811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.415176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=117][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.431984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=100][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.442630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.453046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=102][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.463412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.475716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=99][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.486028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.492650] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81060][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-05-31 02:59:40.496378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=130][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.507335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.519761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=113][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.530978] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=605][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.532839] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=82, this=0x2b09131e5470, extra_info="") [2025-05-31 02:59:40.532966] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=98] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.16.135.166:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=5381, avg_group_log_size=489) [2025-05-31 02:59:40.532989] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, append size=5381) [2025-05-31 02:59:40.533005] INFO [LIB] stat (utility.h:1211) [83013][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2b0913f5e050, extra_info="id:1") [2025-05-31 02:59:40.541442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.543841] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-52801536, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:40.545001] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-05-31 02:59:40.545060] INFO [COMMON] print_io_status (ob_io_manager.cpp:2731) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4778, iops:12, norm_iops:3, bw:56743, [delay/us]:prepare:12, schedule:795, submit:269, rt:3380, total:4458) [2025-05-31 02:59:40.545079] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:40.545186] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4779, avg_iops:12, avg_bw:56747, [delay/us]: prepare:12, schedule:795, submit:269, device:3380, total:4458) [2025-05-31 02:59:40.545203] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:336, idx_:0, tid_:81748}], io_callback_status=runner 0: thread_id=81748, queue_count=0, ) [2025-05-31 02:59:40.545233] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4053, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:396, total:407) [2025-05-31 02:59:40.545259] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5792960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-31 02:59:40.545292] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4053, [delay/us]: prepare:0, schedule:0, submit:600000000, device:396, total:407) [2025-05-31 02:59:40.545310] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:81573}], io_callback_status=runner 0: thread_id=81573, queue_count=0, ) [2025-05-31 02:59:40.545324] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13ef5d0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:40.545333] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2b08f13eeed0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-05-31 02:59:40.545342] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CHANNEL] device_id=0x2b08f13ee0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:40.545351] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2b08f13ed2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-31 02:59:40.545360] INFO [COMMON] print_server_status (ob_io_manager.cpp:424) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=179kB/s, net_out=112kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-05-31 02:59:40.545381] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [81577][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=399475][2nd=2671]}) [2025-05-31 02:59:40.551709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.561974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.562969] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=44] ====== checkpoint timer task ====== [2025-05-31 02:59:40.564006] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=49] succ to get rec scn(*this={this:0x2b0915004030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1748659848819527001, v:0}, max_applied_start_working_ts_:{val:1748659848819527001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1748658385788079, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1118}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:40.564068] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=60] tx ctx memtable get rec scn(this={ObITable:{this:0x2b0952f88080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1748671203580401, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b0952f88080, snapshot_version:{val:1748671203580401, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-31 02:59:40.564540] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=134] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1748671209934467001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1748671209934467001, v:0}}, }, *this={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-31 02:59:40.564639] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=93] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b0913f55150, is_inited:true, freezing_scn:{val:1748671203317153000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1748671209934467001, v:0}, clog_base_lsn:{lsn:2013143040}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1748588325229144000, v:0}, all_id_meta:{id_meta:[{limited_id:1748678390364653000, latest_log_ts:{val:1748678373736140001, v:0}}, {limited_id:13000001, latest_log_ts:{val:1748666220872310001, v:0}}, {limited_id:170000001, latest_log_ts:{val:1748666369264257001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:11, palf_env_:0x2b090afc2030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1748671209934467001, v:0}, ls_id_={id:1}) [2025-05-31 02:59:40.564703] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=55] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-31 02:59:40.564719] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:1748678373736140001, v:0}) [2025-05-31 02:59:40.564728] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:40.564735] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-31 02:59:40.564750] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:40.564771] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-31 02:59:40.564785] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1748671209934467001, v:0}, checkpoint_scn_in_ls_meta={val:1748671209934467001, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-31 02:59:40.564806] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2013143040}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2013143040}, lsn={lsn:2013143040}) [2025-05-31 02:59:40.564829] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2013143040}, ls->get_ls_id()={id:1}) [2025-05-31 02:59:40.564868] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [81744][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=36] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-31 02:59:40.572577] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=106][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.585096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.591495] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81554][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.591576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81556][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.591607] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81557][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.592093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [81562][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.592158] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=737, get_gts_with_stc_cnt=56, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=56, try_wait_gts_elapse_cnt=0) [2025-05-31 02:59:40.592192] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [81545][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-05-31 02:59:40.593046] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [81560][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.593119] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81553][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.593149] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [81555][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-31 02:59:40.595395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.606313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.616834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.628163] WDIAG [SERVER.OMT] get_tenant_base_with_lock (ob_multi_tenant.cpp:193) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5150] get tenant from omt failed(ret=-5150, tenant_id=1003) [2025-05-31 02:59:40.628234] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-5150] switch tenant fail(tenant_id=1003, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.628337] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.628370] WDIAG [SHARE] switch_to (ob_tenant_base.cpp:601) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5150] switch tenant fail(tenant_id=1004, ret=-5150, lbt()="0x9525fb6 0x1c400370 0x8e15e34 0x1a5fe680 0x1a5ffa48 0x1a5fed66 0x22df8975 0x22df66ee 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.630568] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [85795][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=37287) [2025-05-31 02:59:40.633388] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.634558] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [81742][T1_TimerWK1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-31 02:59:40.644629] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83075][T1_L0_G0][T1][YB42AC1087A7-000636654B6DEA97-0-0] [lt=45][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525429826800, cache_obj->added_lc()=false, cache_obj->get_object_id()=53745, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.646491] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [81744][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=37, get_cur_idx()=29, get_pop_idx()=29) [2025-05-31 02:59:40.647738] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2137498583}], scn_array=[cnt:1, {val:1748678380609820000, v:0}], curr_size=589, accum_size=5970, time_cost=13613) [2025-05-31 02:59:40.647928] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [81924][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=188] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.16.135.166:2882", has_set_deleted:false}, accum_size=5970) [2025-05-31 02:59:40.647997] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [81923][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.16.135.166:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=878, avg_log_gen_to_submit_time=982, avg_log_submit_to_flush_time=10619, avg_log_submit_to_slide_time=10697) [2025-05-31 02:59:40.648049] INFO [PALF] read (log_cache.cpp:101) [81926][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=165] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5970, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-05-31 02:59:40.651171] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [83067][T1_L5_G0][T1][YB42AC10870A-0006366724D8CA9A-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-05-31 02:59:40.652039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.663072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.673368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.677268] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [81744][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=79] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):182, sequence:1}) [2025-05-31 02:59:40.683599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.693726] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=597664, replace_num=15728) [2025-05-31 02:59:40.693809] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [81064][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=79] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-05-31 02:59:40.693995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.704138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.714293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.724531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.735075] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.736828] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [35611][T1_L0_G1][T1][YB42AC10871E-0006366549CC130B-0-0] [lt=75] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.16.135.166:2882", committed size=5381) [2025-05-31 02:59:40.747791] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83086][T1_L0_G0][T1][YB42AC1087A6-000636654C3A3910-0-0] [lt=18][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525429929959, cache_obj->added_lc()=false, cache_obj->get_object_id()=53746, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8c771e8 0x8c694ff 0x8c5e4bc 0x8c57866 0x8c5586d 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.748150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.748686] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=160] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-48607232, global_cache_size=85311488, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2025-05-31 02:59:40.749365] INFO [COMMON] wash (ob_kvcache_store.cpp:355) [81065][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash time detail, (compute_wash_size_time=194, refresh_score_time=286, wash_time=0) [2025-05-31 03:59:40.749370] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:29) [37007][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-05-31 02:59:40.753033] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:486) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1, group_id_=1, token=7) [2025-05-31 02:59:40.763285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.766387] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [83088][T1_L0_G0][T1][YB42AC108714-000636671E6D5152-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=525429948557, cache_obj->added_lc()=false, cache_obj->get_object_id()=53747, cache_obj->get_tenant_id()=1, lbt()="0x9525fb6 0x16d03b8e 0x8e153da 0x106e81ca 0x105d3bc7 0x8c562fb 0x8c442cd 0x11090f79 0x22df655e 0x2b08e17e0ea5 0x2b08e1af3b0d") [2025-05-31 02:59:40.775328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.783235] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:553) [81729][T1_HBService][T1][YB42AC1087A6-00063665512A20E4-0-0] [lt=39] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-05-31 02:59:40.785193] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-31 02:59:40.785242] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=41] temporary file module use no blocks [2025-05-31 02:59:40.785254] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-31 02:59:40.785270] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-31 02:59:40.785285] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [82980][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:525428967202, flush_io_finished_ret_:0, flush_io_finished_round_:19948, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-31 02:59:40.789679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.800844] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [81623][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-31 02:59:40.802358] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=13] ====== tenant freeze timer task ====== [2025-05-31 02:59:40.802390] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1738) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=21] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=31457280, ctx.total_memstore_used_=132120576, ctx.total_memstore_hold_=132120576, memstore_freeze_trigger_limit_=53019020, tenant_id=1) [2025-05-31 02:59:40.802412] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1742) [81705][T1_Occam][T1][YB42AC1087A6-000636654B29FC96-0-0] [lt=14] [TenantFreezer] oldest frozen memtable(list="")